Data synchronization method and device among multi-source databases, medium and equipment

Through the data synchronization method between multi-source databases, incremental change data is captured and processed in real time, and the problem of DB2 database data reading affecting performance in the existing technology is solved, real-time synchronization and unified database processing of multi-source data are realized. Improve the system's real-time analysis capabilities.

CN119003652APending Publication Date: 2024-11-22CGN INTELLECTUAL TECH SHENZHEN CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410976931.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing SAP data analysis tools cannot read data from the DB2 database in real time and without any impact, resulting in resource bottlenecks, insufficient performance and slow response. The offline warehousing architecture of the traditional data middle platform cannot meet the real-time data synchronization needs.

Method used

Using data synchronization between multi-source databases, the incremental change data is captured in real time, converted into standard formats and stored in Kafka message queues, and real-time data calculation and distribution are used by Slipstream or third-party programs to ensure unified data entry and processing. , follow the principle of strict one-time processing end-to-end.

Benefits of technology

Real-time and accurate synchronization of multi-source data is realized, the system's retrieval and analysis capabilities are improved, and the system supports 100 million data-level calculation and response are supported, meeting the real-time data needs of the business.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003652A_ABST
    Figure CN119003652A_ABST
Patent Text Reader

Abstract

The invention relates to a data synchronization method and device among multi-source databases, a medium and equipment. The method comprises the following steps: S10, capturing incremental change data from a plurality of source end databases in real time; and S20, converting the incremental change data into a preset standard data format and storing the incremental change data into Kalfa, wherein the Kalfa is a distributed message queue based on a publishing and subscribing mode. And S30, consuming the real-time data in the Kalfa, and carrying out multi-path data distribution to realize unified storage processing. Multi-source data can be accurately synchronized to a third-party database in real time, unified data warehouse management and second-level calculation and response of hundred million-level data are achieved, and then the retrieval and analysis capacity of the system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization processing, and in particular to a method, device, medium and equipment for synchronizing data between multiple source databases. Background Art

[0002] Most SAP systems (enterprise management solution systems) use DB2 databases or other databases such as SQLserver, Oracle and MySQL, but there are currently no mature tools to obtain DB2 data in the form of reading and analyzing the underlying logs without affecting the performance of the DB2 database. In addition, the current data warehouse in Taichung is a classic offline data warehouse architecture with a data integration cycle of T+1 (the next day, representing non-real time, and T+0 representing real time), which cannot meet business needs.

[0003] There are two ways to implement SAP data analysis: through Hana (in-memory database); through PI interface (a data transmission interface of SAP system) to synchronize data to a third-party database. Both methods have problems such as resource and performance bottlenecks, lack of real-time performance and slow response. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device, medium and equipment for synchronizing data between multiple source databases in view of at least one of the above-mentioned defects.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a data synchronization method between multi-source databases, applied to a multi-source data synchronization device, comprising the following steps:

[0006] S10, capturing incremental change data from multiple source databases in real time;

[0007] S20, converting the incremental change data into a preset standard data format and storing it in Kalfa, where Kalfa is a distributed message queue based on a publish-subscribe mode;

[0008] S30, consuming the real-time data in Kalfa, and distributing multiple data channels to achieve unified warehousing processing.

[0009] In the data synchronization method between multi-source databases of the present invention, step S30 further includes:

[0010] The real-time data in Kalfa is consumed through Slipstream or a third-party program, and the real-time data in Kalfa is calculated and converted into standard SQL or API calls, so as to distribute the real-time data in Kalfa to multiple target locations for unified storage processing; Slipstream is a real-time stream calculation engine.

[0011] In the data synchronization method between multi-source databases of the present invention, before step S10, the method further includes:

[0012] The source database turns on the log option and grants the cloud chain platform data synchronization user authority so that the cloud chain platform can capture the incremental change data of the corresponding source database in real time.

[0013] In the data synchronization method between multi-source databases of the present invention, step S10 includes:

[0014] Receive data synchronization tasks submitted by users from the client, and capture incremental change data in real time from the corresponding source database according to the data synchronization tasks.

[0015] In the data synchronization method between multi-source databases of the present invention, the method further includes:

[0016] Steps S10 to S30 follow the end-to-end strictly once processing principle.

[0017] In the data synchronization method between multiple source databases described in the present invention, the source database includes at least one of a DB2 database, a MySQL database, an Oracle database, and a SQL Server database.

[0018] In the data synchronization method between multi-source databases of the present invention, step S10 includes:

[0019] When the source database is a DB2 database or a SQL Server database, the incremental change data is obtained from a table of the DB2 database or the SQL Server database for storing data changes;

[0020] When the source database is a MySQL database, obtaining the incremental change data from the binary log of the MySQL database;

[0021] When the source database is an Oracle database, the incremental change data is obtained from a data change log or a memory of the Oracle database.

[0022] In addition, the present invention also provides a multi-source data synchronization device, comprising:

[0023] The first module is used to capture incremental change data from multiple source databases in real time;

[0024] The second module is used to convert the incremental change data into a preset standard data format and store it in Kalfa, where Kalfa is a distributed message queue based on a publish-subscribe mode;

[0025] The third module is used to consume the real-time data in Kalfa and perform multi-channel data distribution to achieve unified warehousing processing.

[0026] In addition, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is suitable for being loaded by a processor to execute the steps of the above-mentioned method for synchronizing data between multiple source databases.

[0027] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the above-mentioned method for synchronizing data between multiple source databases by calling the computer program stored in the memory.

[0028] The data synchronization method, device, medium and equipment between multi-source databases implemented in the present invention have the following beneficial effects: the present invention can synchronize multi-source data to a third-party database in real time and accurately, realize unified data warehouse management and second-level calculation and response of billions of data, thereby improving the system retrieval and analysis capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 It is a flowchart of a method for synchronizing data between multiple source databases according to an embodiment of the present invention;

[0031] Figure 2 is a flowchart of a method for synchronizing data between multiple source databases according to some embodiments of the present invention;

[0032] Figure 3 is a schematic diagram of the structure of master-slave server data synchronization in some embodiments of the present invention;

[0033] Figure 4 It is a schematic diagram of the structure of the cloud chain platform of some embodiments of the present invention;

[0034] Figure 5 It is a schematic diagram of the user operation flow of the cloud chain platform in some embodiments of the present invention. DETAILED DESCRIPTION

[0035] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0036] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0037] In order to facilitate the understanding of the present invention, the present invention will be described more fully below, and the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0038] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. It is not necessarily required or implied that there is any such actual relationship or order between these entities or operations.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0040] In one embodiment disclosed in the present invention, reference Figure 1 and Figure 2 The data synchronization method between multi-source databases of this embodiment includes the following steps:

[0041] S10. Capture incremental change data from multiple source databases in real time.

[0042] Specifically, step S10 includes:

[0043] Receive data synchronization tasks submitted by users from the client, and capture incremental change data from the corresponding source database in real time based on the data synchronization tasks.

[0044] S20, convert the incremental change data into a preset standard data format and store it in Kalfa, which is a distributed message queue based on a publish-subscribe model. Optionally, the preset standard data format is a JSON format.

[0045] This embodiment captures incremental data changes in the data source library in real time, converts it into a standard JSON format, and writes it to Kafka. It can provide interactive visual configuration and management functions, allowing users to easily configure and manage real-time data synchronization tasks.

[0046] S30, consumes real-time data in Kalfa and distributes multiple data channels to achieve unified warehousing processing.

[0047] Specifically, step S30 also includes:

[0048] Consume the real-time data in Kalfa through Slipstream or third-party programs, perform calculations on the real-time data in Kalfa, convert it into standard SQL or make API calls, and then distribute the real-time data in Kalfa to multiple target locations for unified storage processing. Slipstream is a real-time stream computing engine.

[0049] That is, the incremental data synchronization task will output the processed data to the specified data storage. The target location can be a database, file system, message queue, etc. The processed data can also be output to multiple data storages to meet different business needs.

[0050] That is to say, Slipstream or other third-party programs can distribute real-time data in multiple ways to ClickHouse (a column-based database used for high-performance computing and analysis scenarios), MySQL, Elasticsearch (ES for distributed search and analysis scenarios), and other general databases for unified storage processing.

[0051] This embodiment can synchronize multi-source data to a third-party database in real time and accurately, realize unified data warehouse management and second-level calculation and response of billions of data, thereby improving the system's retrieval and analysis capabilities.

[0052] In some embodiments, before step S10, the method further includes:

[0053] The source database turns on the log option and grants the Cloud Chain platform data synchronization user permission so that the Cloud Chain platform can capture the incremental change data of the corresponding source database in real time.

[0054] Alternatively, the source database includes but is not limited to a DB2 database, a MySQL database, an Oracle database, a SQL Server database, and the like.

[0055] That is to say, the present invention uses an ETL tool (synchronization tool, a process used to extract, convert and load data from the source to the target end) that integrates streams and batches and includes multiple data sources such as DB2, etc., to create tasks in an online visual way, realize real-time / non-real-time capture of data from four data sources including but not limited to DB2, SQLserver, Oracle and MySQL, and perform stream processing operations such as data cleaning, filtering, calculation and association during the transmission process, so as to synchronize the data to a third-party database in real time and accurately, so as to realize unified data warehouse management and second-level calculation and response of billions of data, thereby improving the system retrieval and analysis capabilities.

[0056] Before the improvement, STL (SAP system synchronization tool, used to import data from SAP system to memory database) synchronized data into HANA, and HANA data was synchronized to Hadoop (distributed system basic framework), MySql or third-party database. If the amount of data reached billions or multi-database distribution was required, problems such as long links, poor performance and low stability would occur. In addition, the operation steps were complicated and error-prone.

[0057] After the improvement, a new generation of online visual data real-time synchronization platform is built based on the core technology of FlinkCDC (data change capture tool). It is deployed in a distributed architecture, which can meet the business scenarios of massive data synchronization and realize real-time synchronization and conversion of massive data. It performs well in incremental synchronization, breakpoint continuation, and full synchronization. It also supports full incremental integrated synchronization, while many other open source solutions cannot support full incremental integrated synchronization. Relying on the ecological advantages of Flink (the processing framework of the big data processing engine), the cloud chain platform provides DataStream API (data stream interface) and SQL API (SQL interface). These APIs provide very powerful data conversion capabilities (because the data formats of the two databases are different, the data needs to be converted before it can be transferred between the two databases). At the same time, it supports real-time data capture of four source databases: DB2, MySQL, SQLserver, and Oracle. It first obtains historical data through JDBC (Java database access interface), and then obtains incremental change data through logs or change tables (data change tables).

[0058] Capturing incremental data based on database logs, taking the MYSQL database cluster as an example, the slave server needs to synchronize the data of the master server. It is this mechanism that enables the master-slave cluster of MYSQL to maintain consistent data. Figure 3 In the MYSQL master-slave cluster, the main process of the master-slave synchronization mechanism is as follows:

[0059] 1) The master (primary node) records the changes (captured incremental change data) in the binary log (binarylog, the log in which the MySQL database records changes).

[0060] 2) The slave node copies the master's binary log events (binary log change transactions, incremental change data) to its own relay log.

[0061] 3) The slave redoes the events in the relay log and reflects the changes in its own data.

[0062] Real-time incremental data synchronization actually follows the above principle, disguising itself as a slave of MYSQL to obtain changes in MYSQL-master.

[0063] Specifically, when the source database is a DB2 database or a SQL Server database, the incremental change data is obtained from a table of the DB2 database or the SQL Server database for storing data changes.

[0064] It is understandable that DB2 is a relational database. The DB2 CDC connector can capture row-level changes in tables in the DB2 database. Its implementation principle is based on the SQL replication capability provided by ASN Capture / Apply agents, and stores changes in tables in the database with capture mode enabled in the specified change table (a table used to store data changes in DB2). The DB2CDC connector first reads the historical data in the table through JDBC (Java Database Connectivity, which is an application program interface used in the Java language to standardize the client's access to the database), and then obtains the incremental change data from the changetable, thereby achieving full incremental synchronization. This embodiment can obtain DB2 change data of various versions. Capture mode refers to the default parameter of the table in DB2. After it is enabled, the changes can be stored in the specified change table.

[0065] SQL Server is a widely used database. The SQL Server CDC connector supports capturing and recording row-level changes that occur in the SQL Server database server. The principle is to use the CDC feature provided by SQL Server itself. The CDC capability provided by SQL Server itself can archive the specified changes in the database to the specified changetables. The SQL Server CDC connector first reads the historical data in the table through JDBC, and then obtains the incremental change data from the change tables, thereby achieving full incremental synchronization.

[0066] When the source database is a MySQL database, obtain incremental change data from the binary log of the MySQL database.

[0067] As you can understand, mysql-binlog is the binary log of the MySQL database, which is used to record the SQL statements (except data query statements) of the user's database operations. If you need to synchronize the content of the master database from the database, you can synchronize it through binlog.

[0068] MySQL binlog is stored in multiple files. To locate a LogEvent, you need to use binlog filename+binlog position. Binlog filename+binlog position refers to locating a logEvent (log transaction, which refers to a transaction that adds, deletes, or modifies data) in a binlog (binarylog) by using the file name (filename) and position (position).

[0069] The MySQL binlog data format is mainly divided into statement-based, row-based, and mixed according to the generation method. Statement-based, row-based, and mixed refer to the binlog log format. The statement-based method means that the Master writes SQL statements into the binlog file, and the slave executes the SQL statements in the binlog file after obtaining the binlog file. The row-based method means that the Master records the changes of each row in the table into the binlog file, and the slave implements these changes line by line after obtaining the binlog file. Mixed means mixing these two methods, that is, when recording logs, the most appropriate method will be selected to record logs.

[0070] When the source database is an Oracle database, obtain incremental change data from the data change log or memory of the Oracle database.

[0071] As you can understand, the Oracle CDC connector supports capturing and recording row-level changes that occur in the Oracle database server. The principle is to use the LogMiner tool provided by Oracle or the native XStream API to obtain change data from Oracle.

[0072] LogMiner is an analysis tool provided by Oracle Database. This tool can parse Oracle Redo log files, thereby parsing the database data change log into change event output. When using LogMiner, Oracle server has strict resource restrictions on the process of parsing log files, so data parsing will be slower for tables with particularly large scales. The advantage is that LogMiner can be used for free. Redo logs are redo logs, which are used to ensure the atomicity and persistence of transactions in the database. They record the changes to all data in the database table. When the server fails and the database is restarted, Redo logs can also be used to restore data to the state before the failure.

[0073] XStream API is an internal interface provided by Oracle Database for Oracle GoldenGate (OGG for short, a comprehensive software package mainly used to implement real-time book data integration and replication between Oracle Database and other heterogeneous databases). Clients can efficiently obtain change events through XStream API. The change data is not obtained from the Redo log file, but directly read from a piece of memory in the Oralce server, eliminating the overhead of writing data to the log file and parsing the log file, which is more efficient, but you need to purchase the Oracle GoldenGate (OGG) license.

[0074] This embodiment can simultaneously capture full / incremental data of four databases: DB2, MySQL, SQLserver, and Oracle, thereby achieving full-incremental synchronization of multi-source data.

[0075] In some embodiments, the method further includes: Steps S10 to S30 follow the end-to-end strictly once processing principle.

[0076] That is to say, the cloud chain platform is based on the flink framework (the processing framework of the big data processing engine, including FlinkCDC), and provides Exactly-Once guarantee (end-to-end consistency, from data reading, engine calculation, to writing to external storage, even if the machine or software fails, it ensures that the data is only processed once, without duplication or loss). It is necessary to ensure that the upstream Source layer, the intermediate computing layer, and the downstream Sink layer (downstream receiving layer) simultaneously meet the end-to-end strict one-time processing. The details are as follows:

[0077] 1) After the CloudChain platform consumes Kafka data, it will start a Kafka transaction and write it to the Kafka partition log normally, but mark it as uncommitted. This is the Pre-commit stage.

[0078] 2) Once all Operators complete their respective Pre-commits, they will initiate a commit operation.

[0079] 3) If any Pre-commit fails, all other Pre-commits must be stopped, and Flink will roll back to the most recent successful Checkpoint.

[0080] 4) When all operators complete their tasks, the Sink receives a Checkpoint barrier, saves the current state, stores the Checkpoint, notifies the JobManager, and submits an external transaction to submit the external checkpoint data.

[0081] 5) JobManager receives notifications from all tasks and sends a confirmation message, indicating that the Checkpoint has been completed. Sink receives the confirmation message from JobManager and officially commits the data for this period.

[0082] 6) The external system (Kafka) closes the transaction and the submitted data can be consumed normally.

[0083] From the above process, we can see that once the Pre-commit is completed, it is necessary to ensure that the commit is also completed. Operators and external systems need to guarantee this. If the commit fails (such as network failure, etc.), the Flink application will crash, and then restart according to the user's restart strategy, and then restart the commit. If the commit cannot be executed smoothly, data loss may occur. Therefore, all operators must finally reach a consensus on the Checkpoint: that is, all operators must determine that the data is either successfully executed or terminated and rolled back. The two-phase commit protocol (2PC) is just to solve distributed transactions, which can be processed strictly once from end to end.

[0084] In one embodiment disclosed in the present invention, the multi-source data synchronization device of this embodiment includes:

[0085] The first module is used to capture incremental change data from multiple source databases in real time.

[0086] The second module is used to convert the incremental change data into a preset standard data format and store it in Kalfa, which is a distributed message queue based on the publish-subscribe model.

[0087] The third module is used to consume real-time data in Kalfa and distribute data in multiple channels to achieve unified warehousing processing.

[0088] In one embodiment, reference Figure 4 The multi-source data synchronization device includes a cloud chain platform. The overall architecture of the cloud chain platform can be divided into three layers: computing platform, scheduling platform, and resource platform. Each layer has corresponding functions, and there is interaction between layers, which conforms to the design principles of high cohesion and low coupling.

[0089] Computing platform: It is directly used by developers and can access various external data sources according to business needs to provide subsequent tasks. After the data source configuration is completed, you can perform data synchronization based on the Flink framework visualization or script operation to submit synchronization tasks, and you can perform multi-dimensional monitoring and alarms on running tasks.

[0090] Scheduling platform: After receiving the task content and configuration from the platform, this layer generates the corresponding DataStream according to the differences between the data source and the write target, and finally converts it into JobGraph (data stream combination). The scheduling platform submits the obtained JobGraph to the corresponding resource platform (Kafka) to complete the task submission.

[0091] Resource platform: Currently, it can connect to multiple different resource clusters and different resource types, such as yarn (Hadoop resource manager) and k8s (containerized application platform).

[0092] In some embodiments, the cloud chain platform is mainly composed of two components, namely JobManager (job management platform, master node) and TaskManager (task management platform, slave node). The architecture follows the Master-Slave architecture design principle, JobManager is the Master node, and TaskManager is the Worker (Slave) node. The communication between all components is through the Akka Framework (open source toolkit framework), including task status and Checkpoint triggering information.

[0093] Client: The client is responsible for submitting tasks to the cluster, building an Akka connection with the JobManager, and then submitting tasks to the JobManager, and obtaining the task execution status by interacting with the JobManager.

[0094] The client can submit tasks using the CLI (no interface mode, command line interface mode) or by using the Flink WebUI (interface mode, front-end operation interface mode). It can also specify the RPC network port (remote procedure call network port) of the JobManager in the application to build an ExecutionEnvironment (execution environment) to submit the Flink application.

[0095] JobManager: Responsible for the scheduling of tasks and resource management of the entire Flink cluster. It obtains submitted applications from the client, and then allocates corresponding TaskSlot resources to the submitted applications based on the usage of TaskSlot (task execution unit) on the TaskManager in the cluster and instructs TaskManager to start the application obtained from the client.

[0096] The JobManager is equivalent to the Master node of the entire cluster, and there is only one active JobManager in the entire cluster, which is responsible for task management and resource management of the entire cluster.

[0097] The JobManager and TaskManager communicate through the Actor System to obtain the task execution status and send the application's task execution status to the client through the Actor System.

[0098] At the same time, during the task execution process, Flink JobManager will trigger the Checkpoint operation. After each TaskManager node receives the Checkpoint trigger instruction, it completes the Checkpoint operation. All Checkpoint coordination processes are completed in Fink JobManager.

[0099] When the task is completed, Flink will feedback the task execution information to the client and release the resources in the TaskManager for the next task submission.

[0100] TaskManager: Equivalent to the Slave node of the entire cluster, responsible for specific task execution and resource application and management of corresponding tasks on each node.

[0101] The client compiles and packages the written Flink application and submits it to the JobManager. The JobManager then assigns tasks to TaskManager nodes with resources based on the resource status of the TaskManagers registered in the JobManager, and then starts and runs the tasks.

[0102] TaskManager receives the tasks to be deployed from JobManager, then uses Slot resources to start Task, establishes a network connection for data access, receives data and starts data processing. At the same time, data interaction between TaskManagers is carried out in the form of data streams.

[0103] In some embodiments, it is also possible to implement online visual management of data synchronization tasks, as follows:

[0104] 1. User function permissions, refer to Figure 5 , the system sets two types of user roles:

[0105] 1) Management staff

[0106] User management menu: add users, manage all user login accounts and other user information.

[0107] Task management menu: approve all tasks to be published, and view task status and running information.

[0108] 2) Operator

[0109] Data source configuration menu: query and manage the data source configuration information added by the current account.

[0110] My Tasks: Query and publish data synchronization tasks added by the current account.

[0111] 2. Data source configuration

[0112] Provides search query function, and lists the result set of data source configuration (cfg_source) added by the current login account, with paginated display.

[0113] Input: Name (fuzzy, ignore case), Database Type (drop-down dictionary).

[0114] Output: name, database type, connection address / IP, port, connection account, connection status, creation time, creator account, last modification time, modifier account, and remarks.

[0115] Before modifying and saving data source data, you need to test the database connection and record the data source status field. The API returns the save status and test status.

[0116] 3. Synchronous task operations

[0117] Provides search query function, and the data synchronization task list result set added by the current login account is displayed in pages. The task name links to the task details window, and the data source name links to the data source details (task_source) window, providing entry for modify, run, and stop operation functions.

[0118] Input: name (fuzzy, ignore case), status (drop-down dictionary).

[0119] Output: name, data source, topic, status, creation time, creator account, operation start time, shutdown time, latest checkpoint, and remarks.

[0120] The operator fills in the task name, selects the data source, specifies the target source topic and fills in the name of the table to be synchronized, and submits to save the data source information (Note: the task name only supports English letters, numbers, - and _, and the data source can only be selected from the data source added by the current login and passed the test).

[0121] 4. Task approval management

[0122] Provides managers with a list of all (excluding draft) data synchronization tasks, with paged display. The task name links to the task details window, and the data source name links to the data source details (task_source) window, providing an entry for the approval operation function.

[0123] Input: name (fuzzy, ignore case), status (drop-down dictionary).

[0124] Output: name, data source, topic, status, creation time, creator account, operation start time, shutdown time, latest checkpoint, and remarks.

[0125] The manager approves the task submitted by the operator. If approved, the task status changes to "Not Run", and if not approved, the task status changes to "Not Approved".

[0126] 5. Start the job

[0127] Click the Run button on the task list page to start the job. CloudChain CDC will generate a jar file, submit the Flink client to the Yarn resource scheduling operation, and record the operation information to the hadoop file. The main scheduling process is as follows:

[0128] 1) After submitting the running task, the client uploads the Flink Jar package and configuration to HDFS (one of the core components of the Hadoop system and the lowest-level distributed storage service of Hadoop).

[0129] 2) Then submit the task to Yarn ResourceManager. ResourceManager allocates Container resources and notifies the corresponding NodeManager to start. ResourceManager is the core function in Yarn and is a global resource manager that performs all processing activities by allocating resources and scheduling tasks.

[0130] 3) ApplicationMaster: After the ApplicationMaster is started, it loads the Flink Jar package and configures the build environment. The ApplicationMaster is the application supervisor, which is started by the framework and is responsible for coordinating and managing the components of a specific application.

[0131] 4) Then start the JobManager, and then ApplicationMaster requests resources from ResourceManager to start TaskManager.

[0132] 5) After ResourceManager allocates Container resources, ApplicationMaster notifies the NodeManager of the node where the resources are located to start TaskManager.

[0133] 6)NodeManager loads the Flink Jar package, configures the build environment, and starts TaskManager.

[0134] 7) After the TaskManager is started, it sends a heartbeat packet to the JobManager and waits for the JobManager to assign tasks to it.

[0135] 6. Operation shutdown

[0136] Click the button on the task list page to stop the job. CloudChain CDC stops the specified applicationId through the yarn client interface, which is equivalent to executing the yarn application-kill xxxxxx command to stop the job.

[0137] Task details tracking: On the task operation page, click the task name link in the task list to pop up the task details window.

[0138] Input: Specify the task ID.

[0139] Output: task name, data source, topic, task status, job creation information, and job running information.

[0140] Display the current task approval list in the task details window.

[0141] The task details window displays the current task running information list, Flink log and Yarn log links, and a new Tab page is opened.

[0142] In some embodiments, the cloud chain platform also has the following functions:

[0143] Data source configuration: Provides operators with the data source for data synchronization. Includes the following modules:

[0144] (1) Data source list

[0145] Provides search query function, and lists the result set of data source configuration (cfg_source) added by the current login account, with paginated display.

[0146] Input: Name (fuzzy, ignore case), Database Type (drop-down dictionary).

[0147] Output: name, database type, connection address / IP, port, connection account, connection status, creation time, creator account, last modification time, modifier account, and remarks.

[0148] (2) Add data source

[0149] Before the backend processes and saves the data source data, it is necessary to test the database and record the data source status field. The API returns the save status and test status.

[0150] (3) Modify the data source

[0151] The data source modification window pops up to modify the data source information. Refer to the Add Data Source page.

[0152] Before modifying the data source data on the backend, you need to test the database and record the data source status field. The API returns the operation status and test status.

[0153] (4) Test data source

[0154] In the data source add and modify window, click the [Test Connection] button, the backend requests the connection configuration data and returns the connection status. Multiple database connection tests are supported on demand (database type drop-down options).

[0155] (5) Delete data source

[0156] A secondary confirmation box pops up to logically delete the data source (the status field is marked as 0, which indicates the deletion status).

[0157] In some embodiments, the CloudChain platform also has a My Tasks section, as follows:

[0158] (1) Task List

[0159] Provides search query function, and the data synchronization task list result set added by the current login account is displayed in pages. The task name links to the task details window, and the data source name links to the data source details (task_source) window, providing entry for modify, run, and stop operation functions.

[0160] Input: name (fuzzy, ignore case), status (drop-down dictionary).

[0161] Output: name, data source, topic, status, creation time, creator account, operation start time, shutdown time, latest checkpoint, and remarks.

[0162] Operation button display logic:

[0163] The task status is Draft, Approval Failed->Modifiable.

[0164] The task status is Not Running -> Runnable.

[0165] The task status is Running->Stoppable.

[0166] (2) Add tasks

[0167] The operator fills in the task name, selects the data source, specifies the target source topic and fills in the name of the table to be synchronized, and submits to save the data source information.

[0168] The task name only supports English letters, numbers, -, and _.

[0169] The data source can only be selected from the data source added by the current login and passed the test.

[0170] Submit and save, save as draft status, submit for approval as pending approval status.

[0171] (3) Modify the task

[0172] Modify the data synchronization task information, save it as a draft or submit it for approval. Refer to the Add Task Interface.

[0173] (4) Run the task

[0174] Submit Flink jobs to Yarn resources, and add checkpoints to resume breakpoints. Record task operation information. After successful submission, the current task status is running.

[0175] (5) Stop the task

[0176] Kill the running job resources and record the task job checkpoint and operation information.

[0177] In some embodiments, the cloud chain platform also has a task management section, as follows:

[0178] (1) Task List

[0179] Provides managers with a list of all (excluding draft) data synchronization tasks, with paged display. The task name links to the task details window, and the data source name links to the data source details (task_source) window, providing an entry for the approval operation function.

[0180] Input: name (fuzzy, ignore case), status (drop-down dictionary).

[0181] Output: name, data source, topic, status, creation time, creator account, operation start time, shutdown time, latest checkpoint, and remarks.

[0182] Operation button display logic:

[0183] The task status is pending approval -> available for approval.

[0184] (2) Task Approval

[0185] Managers approve tasks submitted by operators.

[0186] Change the task status to Not Running via –>

[0187] Failed –>The task status changes to Failed.

[0188] (3) Task details

[0189] On the [My Tasks] and [Task Management] pages, click the task name link in the task list to pop up the task details window.

[0190] (4) Task details

[0191] A pop-up window displays detailed information about the task.

[0192] (5) Approval List

[0193] Display the current task approval list (task_audit table, no paging required) in the task details window.

[0194] (6) Run List

[0195] The task details window displays the current task running information list (task_job table, no paging required), Flink log and Yarn log links, and a new Tab page is opened.

[0196] In some embodiments, the cloud chain platform also has a system function module, as follows:

[0197] (1) User authority control

[0198] The menu directory is controlled by identifying the user role attribute (user_role field).

[0199] The backend API interface requests permissions, verifies login status, and controls cross-role access.

[0200] (2) Dictionary constant configuration interface

[0201] Dictionary constant change operation interface, no operation interface is provided yet, simulate request backend, add, delete and modify dictionary object content.

[0202] Dictionary objects involved: 1) user work domain; 2) database type; 3) task status.

[0203] (3)Yarn cluster interface

[0204] Obtaining information such as resource operation status and logs, and performing operations such as killing assets.

[0205] (4) CloudChain Flink Interface

[0206] Checkpoint value acquisition, breakpoint resumption, and task submission to the Yarn asset cluster are all available.

[0207] In one of the embodiments disclosed in the present invention, the computer-readable storage medium of this embodiment stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps of the data synchronization method between multi-source databases as in the above-mentioned embodiment.

[0208] In one of the embodiments disclosed in the present invention, the computer device of this embodiment includes a memory and a processor, the memory stores a computer program, and the processor executes the steps of the data synchronization method between multi-source databases as in the above embodiment by calling the computer program stored in the memory.

[0209] The computer-readable storage medium of the present invention can be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0210] The processor of the present invention is used to provide computing and control capabilities to support the operation of the entire device. It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0211] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0212] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. A data synchronization method between multi-source databases, applied to a multi-source data synchronization device, characterized in that: The following steps are involved: S10, real-time capture of incremental change data from multiple source databases; S20, converting the incremental change data into a preset standard data format and storing it in Kalfa, where Kalfa is a distributed message queue based on a publish-subscribe mode; S30, consuming the real-time data in Kalfa, and distributing multiple data channels to achieve unified warehousing processing.

2. The method for synchronizing data between multiple source databases according to claim 1, characterized in that: Step S30 also includes: The real-time data in Kalfa is consumed through Slipstream or a third-party program, and the real-time data in Kalfa is calculated and converted into standard SQL or API calls, so as to distribute the real-time data in Kalfa to multiple target locations for unified storage processing; Slipstream is a real-time stream calculation engine.

3. The data synchronization method between multiple source databases according to claim 1, characterized in that: Before step S10, the method further includes: The source database turns on the log option and grants the cloud chain platform data synchronization user authority so that the cloud chain platform can capture the incremental change data of the corresponding source database in real time.

4. The method for synchronizing data between multiple source databases according to claim 1, characterized in that: Step S10 includes: Receive data synchronization tasks submitted by users from the client, and capture incremental change data in real time from the corresponding source database according to the data synchronization tasks.

5. The method for synchronizing data between multiple source databases according to claim 1, characterized in that: The method further includes: Steps S10 to S30 follow the end-to-end strictly once processing principle.

6. The method for synchronizing data between multiple source databases according to claim 1, characterized in that: The source database includes at least one of a DB2 database, a MySQL database, an Oracle database, and a SQL Server database.

7. The method for synchronizing data between multiple source databases according to claim 6, characterized in that: Step S10 includes: When the source database is a DB2 database or a SQL Server database, the incremental change data is obtained from a table of the DB2 database or the SQL Server database for storing data changes; When the source database is a MySQL database, obtaining the incremental change data from the binary log of the MySQL database; When the source database is an Oracle database, the incremental change data is obtained from a data change log or a memory of the Oracle database.

8. A multi-source data synchronization device, characterized in that: include: The first module is used to capture incremental change data from multiple source databases in real time; The second module is used to convert the incremental change data into a preset standard data format and store it in Kalfa, where Kalfa is a distributed message queue based on a publish-subscribe mode; The third module is used to consume the real-time data in Kalfa and perform multi-channel data distribution to achieve unified warehousing processing.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps of the method for synchronizing data between multiple source databases as claimed in any one of claims 1 to 7.

10. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the method for synchronizing data between multiple source databases as claimed in any one of claims 1 to 7 by calling the computer program stored in the memory.

Citation Information

Cited By

  • Method, device and system for dynamically generating query interface based on Elasticsearch search engine

    CN119513428A

  • Data storage method, device, system and storage medium

    CN119739352A

  • Remote multi-source industrial data real-time analysis processing system and method

    CN121166786A