System, method and equipment for synchronizing databases among clusters in real time and storage medium

Abstract

The invention discloses an inter-cluster database real-time synchronization system, method and device and a storage medium. The system comprises a collector, an actuator and a management platform. One management platform is provided with a plurality of collectors which are respectively deployed in different source end database clusters, and one management platform is provided with a plurality of actuators. The method comprises the following steps that after a platform normally runs on line, all components are initialized, and a management platform is responsible for recording state information of all the components in the platform; in the active synchronization mode, the collector actively and continuously monitors data file state information of the source end main library and carries out data extraction and data uploading, and then data delivery is carried out through the actuator; when the management platform detects that the heartbeat of a certain collector stops, the management platform notifies a corresponding actuator to start a passive synchronization mode flow; and after receiving the data, the actuator caches the part of data, and then determines whether to analyze the received data according to the setting information of the configuration file. By optimizing the data synchronization algorithm, enhancing the cache data utilization rate and the like, rapid, accurate and safe data synchronization among different database nodes is realized.

Description

Technical Field

[0001] The present invention relates to the field of database management technology, and in particular to a system, method, device and storage medium for real-time synchronization of databases between clusters. Background Art

[0002] With the rapid development of information technology, database cluster synchronization technology plays a vital role in ensuring data consistency and real-time performance. However, existing database cluster synchronization technologies often face some challenges in efficiency when dealing with real-time data synchronization across nodes.

[0003] Specifically, existing database cluster synchronization technologies usually rely on fixed synchronization modes and data transmission protocols. When dealing with data synchronization issues between database clusters, a common method used in the industry is the "export-import" synchronization method. Although this method can ensure data security, its efficiency is obviously greatly reduced.

[0004] Another commonly used method in the industry is to extract and push database log files. This synchronization method is suitable for most databases that use log files to store data information, and can achieve real-time synchronization. However, as the amount of data increases, this method gradually reveals some shortcomings. For example, a large number of data transmission protocols may encounter performance bottlenecks when processing large-scale data synchronization. In addition, this synchronization method often uses a single communication path when the data propagation process changes, which is difficult to effectively maintain high performance when the number of nodes increases or the data traffic increases. Finally, this synchronization method generally generates and retains a part of the cache data files (such as the trails files generated by Oracle GoldenGate during the synchronization process) in the process of pushing the source data to the target end. This part of the cache data file is usually dedicated, that is, one process will generate a corresponding data file. If multiple synchronization processes synchronize data based on the same data source, then they will generate multiple cache data files. Although the synchronization platform usually performs timed or quantitative cache data cleanup, multiple caches of data from the same data source will still cause a certain degree of resource waste. Summary of the invention

[0005] Purpose of the invention: The present invention provides a system, method, device and storage medium for real-time synchronization of databases between clusters, which realizes fast, accurate and secure data synchronization between different database nodes by optimizing data synchronization algorithms, enhancing cache data utilization, etc.

[0006] Technical solution: The system for real-time synchronization of databases between clusters described in the present invention includes: a collector, an executor and a management platform; a management platform has multiple collectors, which are deployed in different source database clusters respectively, and a management platform has multiple executors.

[0007] Furthermore, the collector checks the data status of the source master database, monitors the data file changes of the source master database, and uploads the changed data of the source master database to the executor.

[0008] Furthermore, the executor receives the data stream uploaded by the collector, caches the data stream information uploaded by the collector, parses the data stream into an execution command recognizable by the target end, sends the execution command to the target end master library for command replay, and sends the cached data file to the target end master library.

[0009] Furthermore, the management platform records the status information of each component within the platform, receives external user commands and gives responses, and schedules executors to achieve load balancing of data synchronization.

[0010] Accordingly, a method for real-time synchronization of databases between clusters includes the following steps:

[0011] Step 1: After the platform is online and running normally, each component is initialized, and the management platform is responsible for recording the status information of each component within the platform;

[0012] Step 2: In active synchronization mode, the collector actively and continuously monitors the data file status information of the source master database and extracts and uploads data, and then sends the data through the executor;

[0013] Step 3: When the management platform detects that the heartbeat of a collector stops, the management platform will notify the corresponding executor to start the passive synchronization mode process;

[0014] Step 4: After receiving the data, the executor first caches a copy of the data, and then decides whether to parse the received data according to the settings in the configuration file; if parsed, the received data is parsed into execution commands, and then the execution commands are sent to the target master library for command replay; if not parsed, the data file is sent directly to the target master library.

[0015] Furthermore, in step 2, in the active synchronization mode, the collector actively and continuously monitors the data file status information of the source master database and extracts and uploads data, and then sends the data through the executor, which specifically includes the following steps:

[0016] Step 21: After the collector goes online, it initiates a registration application to the management platform. After receiving the registration application, the management platform records the source database information carried by the collector and then assigns an internally incremented SEQ_ID to the collector.

[0017] Step 22: After the executor goes online, it initiates a registration application to the management platform. After receiving the registration application, the management platform records the service information carried by the executor and then assigns an internally incremented SEQ_ID to the executor.

[0018] Step 23, when synchronization starts for the first time, the collector receives the command, then collects data from all data files of the current source master database and records the data collection progress, including the collected persistent data offset extract_offset and the collected log data sequence number extract_lsn, and then calls the executor's data upload interface to send the data to the executor. After the sending is successful, the collector receives the receiving progress returned by the executor, including the received persistent data offset cache_offset and the received log data sequence number cache_lsn;

[0019] Step 24: When the collection progress is consistent with the receiving progress, the data collection and uploading of this round is terminated; otherwise, the collector continues to upload incremental data until the collection progress is consistent with the receiving progress;

[0020] Step 25: When it is not the first synchronization, the collector will continuously monitor the Mainest file of the master database or other files that record the data status of the master database. When the data file is monitored to change, the collector will compare the data collection progress recorded by itself with the actual amount of data stored in the current source master database, that is, the persistent data offset offset and the log data sequence number lsn, and then proceed to the next step of data extraction.

[0021] Step 26. After the incremental data extraction is completed, the data is sent to the executor. After the sending is successful, the data collection progress is updated. When the executor receives the data uploaded by the collector, it first caches the data in the specified directory of the disk according to the data source information, records or updates the data source information (collector SEQ_ID, etc.) and cache progress (cached persistent data offset cache_offset and cached log data sequence number cache_lsn), and then returns the cache progress to the collector. From the perspective of the collector, the cache progress is the receiving progress.

[0022] Furthermore, in step 3, in the passive synchronization mode, the executor simulates the handshake logic of the source slave library, and sends a series of interactive commands to the source master library to establish a master-slave connection; after the master-slave connection is established, the executor is in standby state, waiting for data sent by the source master library; when the executor receives the data pushed by the source master library, it first caches the data in the specified directory of the disk according to the data source information, and records or updates the data source information (cluster information of the source master library, etc.) and cache progress (cached persistent data offset cache_offset and cached log data sequence number cache_lsn).

[0023] Furthermore, in step 4, in active synchronization mode, the executor receives the source master database data uploaded by the collector; in passive synchronization mode, the executor receives the data pushed by the source master database; the executor marks the data from different source ends and caches them separately, and the cached data is reused by multiple synchronization tasks to save resource consumption of repeatedly obtaining data from the source end.

[0024] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: cache sharing mechanism. In the process of pushing source data to the target end, the prior art generally generates and retains a part of the cache data file. This part of the cache data file is usually dedicated, that is, one process will generate a corresponding data file, and the cache data cannot be shared between multiple processes or multiple tasks. When a new task has a data synchronization requirement, the execution end needs to obtain it from the source database again. By sharing cache data, the present invention effectively reduces the resource consumption of repeatedly obtaining data from the source database in multi-task situations; multi-task parallel processing mechanism. When the prior art processes multi-task scenarios, it is often inefficient due to the degree of coordination between synchronization components and the balance of loads. The present invention optimizes the coordination logic between synchronization components and optimizes the method of balancing loads, thereby effectively handling the problems of high data concurrency and multi-task concurrency; two database synchronization modes. The two modes can run independently or work in conjunction with each other, thereby having a high degree of adaptability to database cluster environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the system architecture of the present invention.

[0026] Figure 2 It is a schematic diagram of the process of the present invention.

[0027] Figure 3 It is a schematic diagram of the cache sharing mechanism flow of the present invention. DETAILED DESCRIPTION

[0028] like Figure 1As shown, a system for real-time synchronization of databases between clusters includes: a management platform of a data synchronization platform. The management platform does not limit the deployment location, and a synchronization platform has only one management platform. The management platform reserves multiple internal interfaces including registration and login, status query, etc. for the collector and executor to call.

[0029] S102, the collector of the data synchronization platform. The collector is deployed in the source database cluster. A synchronization platform can have multiple collectors, which are deployed in different database clusters.

[0030] S103, executor of the data synchronization platform. The executor has no restrictions on the deployment location, and a synchronization platform can have multiple executors. The executor has an internal interface reserved for receiving data, and the collector can upload the collected data to the executor by calling the interface.

[0031] like Figure 2 As shown, a method for real-time synchronization of databases between clusters includes the following steps:

[0032] Step 1: Initialize the components of the data synchronization platform. The collector and executor initiate a registration or login request to the management platform and report status information.

[0033] Step 2: After the management platform is initialized, it starts sending heartbeats to each collector, receives user commands, and schedules the work of each component according to the user commands.

[0034] Step 3, data collection and data upload process. This part is divided into active synchronization and passive synchronization according to different synchronization modes. In active synchronization, the data collection process refers to the collector extracting full and incremental data by monitoring the source data files. The data upload process refers to the collector sending the extracted data to the corresponding executor. In passive synchronization, there is no data collection process, and the acquisition of executor data depends on the data push of the source master database. Therefore, the task of the executor is to establish and maintain the master-slave connection with the source master database.

[0035] Step 4: Data parsing and data push process. In this part, if the target side needs data parsing, the executor will call the parser to parse the uploaded data into execution commands (such as SQL statements). After the parsing is completed, the execution commands will be replayed in the target side master database. If the target side does not need parsing, the executor will establish a data channel with the target side and send the uploaded data directly to the target side master database.

[0036] Table 1 Data synchronization platform execution status at time T0

[0037] Collector (task number) Source cluster Executor (task number) Target cluster Proxy_1(task 1) A Server_1(task 1) B

[0038] Table 2 Data synchronization platform cache status at time T0

[0039] Actual data volume Collection progress Cache progress lsn=999 extract_lsn=999 cache_lsn=999

[0040] Table 3 Data synchronization platform cache status at time T1

[0041] Actual data volume Collection progress Cache progress lsn=1001 extract_lsn=999 cache_lsn=999

[0042] For example, as shown in Table 1, assuming that the data synchronization platform is running normally at time T0, there is 1 collector (Proxy_1) and 1 executor (Server_1) in working state, and 1 task (task 1) is being executed. At this time, as shown in Table 2, the persistent data file has been synchronized, and the log data synchronization progress is consistent with the actual data volume of cluster A. Assume that the synchronization progress is lsn = extract_lsn = cache_lsn = 999. Assuming that at time T1, cluster A has two consecutive write records (SET K1V1, SET K2 V2), then the actual data volume of cluster A will increase, and after the increase, lsn = 1001. At this time, the status is shown in Table 3. Proxy_1 finds that the data has changed, and it will extract the incremental data of the change, that is, extract the data corresponding to the lsn values ​​​​of 1000 and 1001. Then send this part of the incremental data to Server_1. After receiving the data, Server_1 returns the value of cache_lsn to Proxy_1 after successful caching. Proxy_1 compares extract_lsn with cache_lsn to ensure that the data push is successful. Server_1 calls the parser to parse the received data into execution commands "SET K1 V1" and "ST K2V2", and finally replays "SET K1 V1" and "SET K2 V2" in the master database of cluster B.

[0043] like Figure 3 As shown in FIG. 1 , the specific process of the cache sharing mechanism in synchronization work includes the following steps:

[0044] Step 1: Executor selection process. When a new synchronization task is launched, the management platform will sort the executors currently executing synchronization tasks according to their own task load, and then select the executor with the least task load to undertake the new task. If the task load of all executors has reached the upper limit, the management platform will pull up a new executor to undertake the new task.

[0045] Step 2: Query process of cache status. The management platform obtains the source database information by parsing the synchronization task command, and retrieves the cache list based on the data source information. If the current data source is found to have a cache, the management platform obtains the cached data volume through the internal interface provided by the executor and tells the executor to turn off the cache function of the task. If the current data source is found to have no cache, the management platform will initialize the synchronization data volume.

[0046] Step 3: Command sending process: The management platform obtains the amount of cached data and then sends it to different components according to different synchronization modes.

[0047] Table 4 Execution of data synchronization platform at time T0

[0048] Source database cluster Data synchronization platform executor (task number) Target database cluster A Server_1(task 1) D B Server_2(task 2) E C Server_2(task 3) F

[0049] Table 5 Data synchronization platform cache status at time T0

[0050] Source cluster Platform internal marking Cached data files A Source_000001 metadata.dbf,dump.rdb,vlog.dbf B Source_000002 metadata.dbf,dump.rdb,vlog.dbf C Source_000003 metadata.dbf,dump.rdb

[0051] Table 6 Execution of the data synchronization platform at time T1

[0052] Source database cluster Data synchronization platform executor (task number) Target database cluster A Server_1(task 1) D B Server_2(task 2) E C Server_2(task 3) F A Server_1(task 4) G

[0053] For example, as shown in Table 4, it is assumed that at time T0, the data synchronization platform already has two executors working (Server_1 and Server_2), and is executing three tasks (task 1, task 2, and task 3). At this time, a new task, task 4, is launched. This task needs to synchronize the data of cluster A to cluster G. Then, according to S310, the management platform selects Server_1 to undertake task 4. Then, as shown in Table 5, the management platform retrieves the cache status and finds that the source end cluster A of task 4 already has a cache. Assuming that the cache progress is cache_lsn=999, the management platform then notifies Server_1 to start executing task 4, pushes the cached data to the target end, and notifies Server_1 to turn off the cache function of task 4. After Server_1 completes the data push of task 4, it notifies the management platform. After receiving the notification, the management platform notifies the collector on cluster A of the cache progress and the information of Server_1 (task 4), so that it starts working on task 4. The execution status of the synchronization platform is shown in Table 6.

Claims

1. A system for real-time synchronization of databases between clusters, characterized in that: include: Collector, executor and management platform; A management platform has multiple collectors, which are deployed in different source database clusters, and a management platform has multiple executors.

2. The system for real-time synchronization of databases between clusters according to claim 1, characterized in that: The collector checks the data status of the source master database, monitors the data file changes of the source master database, and uploads the changed data of the source master database to the executor.

3. The system for real-time synchronization of databases between clusters as claimed in claim 1, characterized in that: The executor receives the data stream uploaded by the collector, caches the data stream information uploaded by the collector, parses the data stream into execution commands recognizable by the target end, sends the execution commands to the target end master library for command replay, and sends the cached data files to the target end master library.

4. The system for real-time synchronization of databases between clusters as claimed in claim 1, characterized in that: The management platform records the status information of each component within the platform, receives and responds to external user commands, and schedules executors to achieve load balancing for data synchronization.

5. A method applied to the system for real-time synchronization of inter-cluster databases as claimed in claim 1, characterized in that: The steps include: Step 1: After the platform is online and running normally, each component is initialized, and the management platform is responsible for recording the status information of each component within the platform; Step 2: In active synchronization mode, the collector actively and continuously monitors the data file status information of the source master database and extracts and uploads data, and then sends the data through the executor; Step 3: When the management platform detects that the heartbeat of a collector stops, the management platform will notify the corresponding executor to start the passive synchronization mode process; Step 4: After receiving the data, the executor first caches a copy of the data, and then decides whether to parse the received data according to the settings in the configuration file; if parsed, the received data is parsed into execution commands, and then the execution commands are sent to the target master database for command replay; If not parsed, the data file will be sent directly to the target master database.

6. The method for real-time synchronization of databases between clusters as claimed in claim 5, characterized in that: In step 2, in active synchronization mode, the collector actively and continuously monitors the data file status information of the source master database and extracts and uploads data, and then sends the data through the executor. The specific steps include: Step 21: After the collector goes online, it initiates a registration application to the management platform. After receiving the registration application, the management platform records the source database information carried by the collector and then assigns an internally incremented SEQ_ID to the collector. Step 22: After the executor goes online, it initiates a registration application to the management platform. After receiving the registration application, the management platform records the service information carried by the executor and then assigns an internally incremented SEQ_ID to the executor. Step 23, when synchronization starts for the first time, the collector receives the command, then collects data from all data files of the current source master database and records the data collection progress, including the collected persistent data offset extract_offset and the collected log data sequence number extract_lsn, and then calls the executor's data upload interface to send the data to the executor. After the sending is successful, the collector receives the receiving progress returned by the executor, including the received persistent data offset cache_offset and the received log data sequence number cache_lsn; Step 24: When the collection progress is consistent with the receiving progress, the data collection and uploading of this round is terminated; otherwise, the collector continues to upload incremental data until the collection progress is consistent with the receiving progress; Step 25: When it is not the first synchronization, the collector will continuously monitor the Mainest file of the master database or other files that record the data status of the master database. When the data file is monitored to change, the collector will compare the data collection progress recorded by itself with the actual amount of data stored in the current source master database, that is, the persistent data offset offset and the log data sequence number lsn, and then proceed to the next step of data extraction. Step 26. After the incremental data extraction is completed, the data is sent to the executor. After the sending is successful, the data collection progress is updated. When the executor receives the data uploaded by the collector, it first caches the data in the specified directory of the disk according to the data source information, records or updates the data source information and cache progress, and then returns the cache progress to the collector. From the perspective of the collector, the cache progress is the receiving progress.

7. The method for real-time synchronization of databases between clusters as claimed in claim 5, characterized in that: In step 3, in the passive synchronization mode, the executor simulates the handshake logic of the source slave library and sends a series of interactive commands to the source master library to establish a master-slave connection. After the master-slave connection is established, the executor is in standby state, waiting for data sent by the source master library. When the executor receives the data pushed by the source master library, it caches the data in the specified directory of the disk according to the data source information, and records or updates the data source information and cache progress.

8. The method for real-time synchronization of databases between clusters as claimed in claim 5, characterized in that: In step 4, in active synchronization mode, the executor receives the source master database data uploaded by the collector; in passive synchronization mode, the executor receives the data pushed by the source master database; the executor marks the data from different source ends and caches them separately. The cached data is reused by multiple synchronization tasks to save resource consumption of repeatedly obtaining data from the source end.

9. A device for real-time synchronization of databases between clusters, characterized in that: include: one or more processors; A storage device for storing one or more programs and user data; When the one or more programs are executed by one or more processors, the one or more processors implement the method for real-time synchronization of databases between clusters as described in any one of claims 5 to 8.

10. A storage medium for real-time synchronization of databases between clusters, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method for real-time synchronization of databases between clusters as claimed in any one of claims 5 to 8 is implemented.

Images