Database bidirectional synchronization method and computer program product

Abstract

The invention discloses a database bidirectional synchronization method and a computer program product, and relates to the technical field of databases, and the method comprises the steps that a source end database cluster receives externally inserted data change operation, updates a source data table and writes the source data table into a first archiving log to obtain a second archiving log, determining a data change record corresponding to the data change operation from the second archiving log and reporting the data change record to an acquisition processor system; the acquisition processor system receives the data change record, determines a target data table having a synchronization relationship with the source data table, and sends the data change record to a target database cluster where the target data table is located; and the target database cluster performs data synchronization operation on the target data table according to the data change record, and writes the data synchronization operation into a third archiving log. According to the method, the synchronization flow direction of the data change among the multiple clusters is controlled by taking the data table as a basic unit, so that the performance loss and redundant space occupation in the data synchronization process are reduced, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database bidirectional synchronization method and a computer program product. Background Art

[0002] In the prior art, the distributed database relies on a data change collector to collect data changes from the logical replication slots of the data nodes of the distributed database cluster, and sends the collected results to a data change processor, which implements DTPC (distributed transaction plane control) and finally outputs them to another database cluster to achieve the transaction consistency function of distributed data synchronization. However, in the existing synchronization scheme, the database cluster cannot act as both a publisher and a subscriber of synchronized data. If two-way synchronization is required, it is necessary to perform two-way loop control filtering in the database external tool, but this method has the problems of large performance loss and low synchronization efficiency in scenarios with large data volumes.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a database bidirectional synchronization method and a computer program product, aiming to solve the technical problems of large performance loss and low synchronization efficiency when performing data synchronization in scenarios with large data volumes.

[0005] To achieve the above object, the present application proposes a database bidirectional synchronization method, which is applied to an acquisition processor system and includes:

[0006] Receiving data change records sent by source-end database clusters in each distributed database cluster, wherein the source-end database cluster receives an externally inserted data change operation, updates a source data table according to the data change operation, writes the data change operation into a first archive log to obtain a second archive log, determines a data change record corresponding to the data change operation from the second archive log based on a preset table-level customization rule and a preset synchronization task, and reports the data change record to the acquisition processor system;

[0007] Based on a preset table-level distribution rule, a target data table having a synchronization relationship with the source data table is determined, and a target database cluster where the target data table is located is determined from each of the distributed database clusters, and the data change record is sent to the target database cluster, wherein the target database cluster performs a data synchronization operation on the target data table according to the data change record, and writes the data synchronization operation into a third archive log.

[0008] In one embodiment, the database bidirectional synchronization method includes:

[0009] Obtaining the cluster identity key of each of the distributed database clusters, generating a synchronization task based on each of the cluster identity keys, and sending the synchronization task to each of the distributed databases, respectively, wherein the distributed database determines the received synchronization task as a preset synchronization task;

[0010] Generating respective table-level customization rules for each of the distributed database clusters, and sending each of the table-level customization rules to the corresponding distributed databases, wherein the distributed databases determine the received table-level customization rules as preset table-level customization rules;

[0011] A streaming transmission channel with each of the distributed database clusters is created based on each of the cluster identity keys, wherein synchronization data is transmitted through the streaming transmission channel.

[0012] In one embodiment, the step of obtaining the cluster identity key of each of the distributed database clusters includes:

[0013] The system identity information of the acquisition processor system is sent to each of the distributed database clusters respectively, wherein the global transaction manager in the distributed database cluster obtains the MAC address and the random number, and concatenates the MAC address and the random number to obtain an initial identifier, encodes the initial identifier to obtain a cluster identifier of the distributed database cluster, concatenates the acquisition processor system identity information and the cluster identifier to obtain concatenated information, encrypts the concatenated information through a preset key file to obtain a cluster identity key, and sends the cluster identity key to the acquisition processor system;

[0014] Receive a cluster identity key sent by each of the distributed database clusters.

[0015] In one embodiment, the step of generating table-level customization rules for each of the distributed database clusters includes:

[0016] Traversing each of the distributed database clusters, determining a first data entry with the distributed database cluster as a data flow source from a preset flow direction relationship table, wherein the flow direction relationship table is used to characterize the data flow direction relationship between each data table in each of the distributed database clusters;

[0017] Determine, from the first data entry, a second data entry with the distributed database cluster as a data flow target end;

[0018] The second data entry is filtered out from each data entry in the flow relationship table to obtain a third data entry, a data table in the third data entry is determined as a synchronization data table of the distributed database cluster, and data table identifiers of each synchronization data table are aggregated to obtain a table-level customization rule of the distributed database cluster.

[0019] To achieve the above objectives, the present application proposes a database bidirectional synchronization method, which is applied to any one of the distributed database clusters, and includes:

[0020] If the distributed database cluster is a source-end database cluster, the source data table is updated according to the received external insert data change operation, the data change operation is written into a first archive log to obtain a second archive log, and a data change record corresponding to the data change operation is determined from the second archive log based on a preset table-level customization rule and a preset synchronization task, and the data change record is reported to the acquisition processor system, wherein the acquisition processor system determines a target data table having a synchronization relationship with the source data table based on a preset table-level distribution rule, and determines a database cluster where the target data table is located from each of the distributed database clusters, and sends the data change record to the database cluster;

[0021] If the distributed database cluster is a target database cluster, a data synchronization operation is performed on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and the data synchronization operation is written into a third archive log.

[0022] In one embodiment, the database bidirectional synchronization method further includes:

[0023] Parsing the preset synchronization task to obtain the cluster identity key and the first system identity information of each of the distributed database clusters;

[0024] Each of the cluster identity keys is traversed, and the cluster identity key is decoded to obtain a cluster identifier and second system identity information. If the first system identity information is consistent with the second system identity information, the cluster identifier is saved.

[0025] In one embodiment, the step of determining the data change record corresponding to the data change operation from the second archive log based on the preset table-level customization rule and the preset synchronization task, and reporting the data change record to the acquisition processor system includes:

[0026] Parsing the second archive log by a preset synchronization task to obtain a plurality of parsed data change records, and filtering out data change records carrying non-current cluster identifiers from each of the parsed data change records to obtain candidate data change records, wherein the non-current cluster identifier is a cluster identifier of a database cluster other than the source database cluster in each of the distributed database clusters;

[0027] Based on a preset table-level customization rule, data change records of a non-synchronized data table are screened out from the candidate data change records to obtain a first data change record.

[0028] In one embodiment, before the step of performing a data synchronization operation on a target data table corresponding to the second data change record received from the acquisition processor system and writing the data synchronization operation into a third archive log, the step further includes:

[0029] receiving third system identity information, and decrypting the source cluster identity key set on the first connection session to obtain a source cluster identity identifier and fourth system identity information, wherein the acquisition processor system creates the first connection session with the target database cluster and sets the source cluster identity key on the first connection session;

[0030] If the third system identity information is consistent with the fourth system identity information, the source cluster identifier is bound to the first connection session to obtain a first identification session, and the source cluster identifier is bound to the second connection session to obtain a second identification session, wherein the second connection session is a connection session between the coordinator node and each data node in the target database cluster.

[0031] In one embodiment, the step of performing a data synchronization operation on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and writing the data synchronization operation into a third archive log includes:

[0032] Receiving, through the coordinator node, a data change operation statement based on the first identification session, wherein the acquisition processor system parses the second data change record into the data change operation statement and sends it to the coordinator node based on the first identification session;

[0033] Sending the data change operation statement to the data node where the target data table is located based on the second identification session through the coordinator node, and executing the data change operation statement in the target data table;

[0034] A log record is generated based on the data change operation statement and the source cluster identifier, and the log record is stored in a third archive log.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the database bidirectional synchronization method described above are implemented.

[0036] In the present application, a source database cluster receives an externally inserted data change operation, updates a source data table according to the data change operation, writes the data change operation into a first archive log to obtain a second archive log, determines a data change record corresponding to the data change operation from the second archive log based on a preset table-level customization rule and a preset synchronization task, and reports the data change record to an acquisition processor system; the acquisition processor system receives data change records sent by source database clusters in each distributed database cluster, determines a target data table having a synchronization relationship with the source data table based on a preset table-level distribution rule, determines a target database cluster where the target data table is located from each distributed database cluster, and sends the data change record to the target database cluster; the target database cluster performs a data synchronization operation on the target data table according to the data change record, and writes the data synchronization operation into a third archive log.

[0037] Compared with data synchronization processing based on databases, this application uses data tables as the basic units to control the synchronization flow of data changes among multiple clusters, avoiding synchronization of private data tables of database clusters during database synchronization, reducing synchronization of invalid data, thereby reducing performance loss and excess space usage during data synchronization and improving data synchronization efficiency.

[0038] At the same time, during each synchronization, table-level customized rules and synchronization tasks are used to screen out the data change records corresponding to the data change operations inserted outside the database cluster, and then synchronize the data change records, thereby realizing the synchronization of externally inserted data change operations, avoiding the distributed database cluster from synchronizing the data change operations received from other database clusters again, avoiding the loop conflict problem of the distributed database, and improving the security of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of the synchronization process of the existing distributed database involved in this application;

[0042] Figure 2 A flowchart of the first embodiment of the bidirectional database synchronization method of the present application is provided;

[0043] Figure 3 A timing diagram of a database bidirectional synchronization method provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a timing sequence for generating a cluster identity key provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a process for storing a cluster identity provided in an embodiment of the present application;

[0046] Figure 6 A timing diagram of synchronous data change provided in an embodiment of the present application;

[0047] Figure 7 A timing diagram of creating a streaming transmission channel provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of a synchronization data table provided in accordance with an embodiment of the present application;

[0049] Fig. 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the database bidirectional synchronization method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: the source database cluster receives an externally inserted data change operation, updates the source data table according to the data change operation, writes the data change operation into a first archive log to obtain a second archive log, determines the data change record corresponding to the data change operation from the second archive log based on a preset table-level customization rule and a preset synchronization task, and reports the data change record to the acquisition processor system; the acquisition processor system receives the data change record sent by the source database cluster in each distributed database cluster, determines the target data table having a synchronization relationship with the source data table based on a preset table-level distribution rule, and determines the target database cluster where the target data table is located from each of the distributed database clusters, and sends the data change record to the target database cluster; the target database cluster performs a data synchronization operation on the target data table according to the data change record, and writes the data synchronization operation into a third archive log.

[0054] Please refer to Figure 1 The existing synchronization process of distributed databases is as follows: the data change collector collects data changes from the logical replication slots of the data nodes of the distributed database cluster, sends the collected results to a data change processor, and the processor implements DTPC (distributed transaction plane control), and finally outputs them to the database cluster B to achieve the transaction consistency function of distributed data synchronization. However, in the existing synchronization scheme, the database cluster cannot act as both a synchronization data publisher and a synchronization data subscriber. Analyzing the current solutions for two-way synchronization, most of them involve stand-alone databases, or filter processing of two-way loop control in database external tools, which has certain reference value.

[0055] In addition, the loop control of a stand-alone database mainly relies on the publish-subscribe function or external tools to filter data changes by copying the source. However, in order to avoid loop conflicts, the synchronization solution of a distributed database needs to consider the consistency of transactions. Different from the solution of a stand-alone database in the synchronization architecture, a data change processor is required to handle consistency issues. If a filtering solution for loop control in an external database tool is adopted, the string obtained by processing the logical replication decoding needs to be filtered during the filtering process. In this way, there will be additional performance loss in the process of processing the string, which is particularly obvious in scenarios with large data volumes. In the current loop control solution, data changes are mainly synchronized based on the database as the basic unit. However, in many actual scenarios, users need to control the synchronization flow of data changes between multiple clusters based on the data table as the basic unit. However, many data tables are private to the cluster and do not participate in synchronization. If too much invalid data is synchronized, it will inevitably cause performance loss and redundant space occupation, greatly reducing synchronization efficiency.

[0056] The present application provides a solution. Compared with data synchronization processing based on databases, the present application uses data tables as basic units to control the synchronization flow of data changes among multiple clusters, avoiding synchronization of private data tables of database clusters during database synchronization, reducing synchronization of invalid data, thereby reducing performance loss and excess space occupation during data synchronization and improving data synchronization efficiency.

[0057] At the same time, during each synchronization, table-level customized rules and synchronization tasks are used to screen out the data change records corresponding to the data change operations inserted outside the database cluster, and then synchronize the data change records, thereby realizing the synchronization of externally inserted data change operations, avoiding the distributed database cluster from synchronizing the data change operations received from other database clusters again, avoiding the loop conflict problem of the distributed database, and improving the security of data synchronization.

[0058] It should be noted that the execution subject of this embodiment can be a data change acquisition processor system (hereinafter referred to as the acquisition processor system for convenience of description), and the acquisition processor system is deployed on a computing service device with data processing, network communication and program running functions. The following describes this embodiment with the acquisition processor system as the execution subject.

[0059] Based on this, the present application embodiment provides a database bidirectional synchronization method, referring to Figure 2 , Figure 2 This is a flowchart of the first embodiment of the database bidirectional synchronization method of the present application.

[0060] In this embodiment, the database bidirectional synchronization method includes steps S10 to S20:

[0061] Step S10, receiving data change records sent by source-end database clusters in each distributed database cluster, wherein the source-end database cluster receives externally inserted data change operations, updates the source data table according to the data change operations, writes the data change operations into a first archive log to obtain a second archive log, determines the data change records corresponding to the data change operations from the second archive log based on preset table-level customization rules and preset synchronization tasks, and reports the data change records to the acquisition processor system;

[0062] It should be noted that a distributed database cluster is a system composed of multiple database nodes, which are interconnected through a network and work together to provide data storage and access services. It includes at least a global transaction manager node, a coordinator node, and a data node. The global transaction manager node is responsible for coordinating and managing transactions in the entire cluster to ensure the consistency and integrity of transactions; the coordinator node is responsible for processing client requests, distributing the requests to appropriate data nodes, and collecting processing results and returning them to the client; the data node is the node that actually stores data and is responsible for data reading and writing operations. The data node can independently perform database operations and communicate with other data nodes to achieve distributed storage and processing of data. In a distributed database system, shards or partitions of a data table can be stored across multiple data nodes to achieve load balancing and performance optimization. When a user needs to access or modify data in a data table, the database system will route the request to the corresponding data node for processing based on the distribution of the data.

[0063] The collection processor system is a system used to collect and process data changes in distributed database clusters. The system collects data change records from the data nodes of each distributed database cluster, and then sends these records to the data change processor for summary processing and transaction control operations, and finally synchronizes the data changes to the target database cluster.

[0064] The source database cluster is the database cluster in each distributed database cluster that receives externally inserted data change operations. Externally inserted data change operations refer to operations that insert or update data into a distributed database cluster by a business client or other external system. These operations will cause the content of the data table in the database cluster to change and trigger the data synchronization process.

[0065] In this embodiment, the source database cluster updates the corresponding data table (i.e., the source data table) according to the data change operation, writes the data change operation into the first archive log to obtain the second archive log, determines the data change record corresponding to the data change operation from the second archive log based on the preset table-level customization rules and the preset synchronization task, and reports the data change record to the acquisition processor system. Among them, the table-level customization rules refer to the rules used to control the flow of data tables between multiple clusters in the synchronization system. Specifically, the source cluster and target cluster of the data table that needs to be concerned can be configured on the data change acquisition processor system; the synchronization task refers to the task sent by the data change acquisition processor system to the distributed database cluster for realizing data synchronization; the data change record refers to the detailed information of the data change in the source data table, including the type of data change (such as insert, update, delete, etc.), the content of the changed data, and the time when the change occurred, which is the basis of data synchronization. The acquisition processor system receives the data change records sent by the source database cluster in each distributed database cluster.

[0066] Step S20, based on the preset table-level distribution rule, determine the target data table that has a synchronization relationship with the source data table, and determine the target database cluster where the target data table is located from each distributed database cluster, and send the data change record to the target database cluster, wherein the target database cluster performs data synchronization operations on the target data table according to the data change record, and writes the data synchronization operations into the third archive log.

[0067] The acquisition processor system determines the target data table having a synchronization relationship with the source data table based on the preset table-level distribution rules, determines the target database cluster where the target data table is located from each distributed database cluster, and sends the data change record to the target database cluster.

[0068] The target database cluster performs data synchronization operations on the target data table according to the data change record, and writes the data synchronization operations into the third archive log.

[0069] In one feasible implementation, the database bidirectional synchronization method includes:

[0070] Step S01, obtaining the cluster identity key of each distributed database cluster, generating a synchronization task based on each cluster identity key, and sending the synchronization task to each distributed database respectively, wherein the distributed database determines the received synchronization task as a preset synchronization task;

[0071] Before data synchronization, each distributed data is configured to complete the initialization of data synchronization, providing necessary configuration and preparation for subsequent data synchronization. The initialization operation may include: configuring synchronization tasks and table-level customization rules for the distributed database cluster, establishing a streaming channel between the acquisition processor system and the distributed database cluster, and in addition, the initialization operation may also include key file synchronization, system resource allocation, log configuration, error handling and other operations, which are not limited here and can be set according to actual needs.

[0072] Specifically, the acquisition processor system obtains the cluster identity key of the distributed database cluster and generates a synchronization task based on the key, ensuring the security and accuracy of the synchronization task. The distributed database determines the received synchronization task as a preset synchronization task to perform subsequent data synchronization processing based on the preset synchronization task. It should be noted that the cluster identity key is a key element of the security authentication and loopback control mechanism used in the bidirectional synchronization process of a distributed database. It is generated by the global transaction manager node of the distributed database cluster and contains the unique identifier of the cluster and the system identity information of the acquisition processor system. In a specific implementation method, the cluster identity key can be encrypted by a specific encryption algorithm (such as AES128) and stored and transmitted in Base64 encoding to ensure its security and uniqueness.

[0073] Step S02, generating table-level customization rules for each distributed database cluster, and sending each table-level customization rule to each corresponding distributed database, wherein the distributed database determines the received table-level customization rule as a preset table-level customization rule;

[0074] The collection processor system generates table-level customization rules for each distributed database cluster, and sends each table-level customization rule to each corresponding distributed database. The distributed database determines the received table-level customization rule as a preset table-level customization rule for subsequent data synchronization processing.

[0075] Step S03: creating streaming transmission channels with each distributed database cluster based on each cluster identity key, wherein synchronization data is transmitted through the streaming transmission channels.

[0076] The collection processor system creates streaming transmission channels with each distributed database cluster based on the identity keys of each cluster. The streaming transmission channel is a technical means for real-time transmission of data change records during the two-way synchronization of distributed databases. It allows a persistent, two-way data transmission path to be established between the data change collection processor system and the database cluster, thereby achieving real-time synchronization and updating of data.

[0077] It should be noted that in the process of establishing a streaming transmission channel, an authentication operation can be performed based on the cluster identity key to ensure the security of the transmission channel. The process of establishing a streaming transmission channel can be: when the data synchronization task is started, the acquisition processor system creates a connection session to each data node of the database cluster and sends an IDENTIFY_REPLSYNC command (identification replication synchronization command), which carries the cluster identity key and system identity information of the cluster to which the data node belongs; the data node parses the cluster identity key to obtain the system identity information contained in the key (hereinafter referred to as the reference identity information for distinction), and if the reference identity information is consistent with the system identity information carried in the command, it is determined that the verification is successful; in the case of successful verification, the acquisition processor system establishes a streaming transmission channel with the distributed database cluster. When the business client performs a data change operation (such as inserting, updating or deleting data) on the distributed database cluster, the change will be captured in real time and parsed into an archive log. Then, the acquisition processor system synchronizes the change to the target database cluster through the streaming transmission channel, thereby realizing two-way synchronization and update of data.

[0078] In one feasible implementation, step S01: the step of obtaining the cluster identity key of each distributed database cluster includes:

[0079] Step S011, sending the system identity information of the acquisition processor system to each distributed database cluster respectively, wherein the global transaction manager in the distributed database cluster obtains the MAC address and the random number, and concatenates the MAC address and the random number to obtain an initial identifier, encodes the initial identifier to obtain a cluster identifier of the distributed database cluster, concatenates the acquisition processor system identity information and the cluster identifier to obtain concatenated information, encrypts the concatenated information through a preset key file to obtain a cluster identity key, and sends the cluster identity key to the acquisition processor system;

[0080] The acquisition processor system sends system identity information to each distributed database cluster. The system identity information may include system name, version number, serial number, etc., which is used to identify and verify the identity of the acquisition processor system.

[0081] The global transaction manager in the distributed database cluster obtains the MAC address from the network interface card (NIC) of the cluster and generates a random number. The MAC address is a unique identifier at the hardware level, and the random number increases the randomness and security of the identifier. The MAC address and the random number are concatenated to form an initial identifier, and the initial identifier is encoded (such as hashing, encryption, etc.) to obtain the cluster identifier of the distributed database cluster. The encoding process is designed to ensure the uniqueness and security of the identifier. The global transaction manager concatenates the identity information of the acquisition processor system with the cluster identifier to form concatenated information, and uses the preset key file to encrypt the concatenated information to obtain the cluster identity key. The global transaction manager sends the generated cluster identity key back to the acquisition processor system. It should be noted that the key file is a known and secure encryption key source, which is used to ensure the security of the encryption process. In a feasible implementation, the global transaction manager node uses the gs_guc generate command to generate the key files of the synchronization system: replsync.key.cipher ciphertext file and replsync.key.rand encryption factor file. After the distributed database cluster is initialized for the first time, the key file is sent to all nodes of all clusters in the synchronization system for subsequent use of the key file for encryption and decryption processing.

[0082] The acquisition processor system sends the system identity information of the acquisition processor system to each distributed database cluster respectively.

[0083] Step S012: receiving a cluster identity key sent by each distributed database cluster.

[0084] The acquisition processor system receives the cluster identity keys sent by each distributed database cluster and stores these keys in a secure storage medium (such as an encrypted database or file system) for subsequent use. At the same time, the acquisition processor system will also verify the received keys to ensure their integrity and correctness. The verification process may include checking whether the key format, length, encryption method, etc. meet expectations.

[0085] In one feasible implementation, step S02: generating table-level customization rules for each distributed database cluster, includes:

[0086] Step S021, traversing each distributed database cluster, and determining a first data entry with the distributed database cluster as a data flow source from a preset flow direction relationship table, wherein the flow direction relationship table is used to characterize the data flow direction relationship between each data table in each distributed database cluster;

[0087] Traverse each distributed database cluster and determine the first data entry with the distributed database cluster as the data flow source from the preset flow relationship table. It should be noted that the flow relationship table is a predefined data structure used to record the data flow relationship between data tables in each distributed database cluster. The data flow table may include information such as the source database cluster, the target database cluster, and the data tables flowing between them.

[0088] Step S022, determining a second data entry with the distributed database cluster as a data flow target end from the first data entry;

[0089] Among the filtered first data entries, a data entry with the currently traversed distributed database cluster as the data flow target end is searched, that is, the second data entry.

[0090] Step S023, filter out the second data entry from each data entry in the flow relationship table to obtain a third data entry, determine the data table in the third data entry as the synchronization data table of the distributed database cluster, and aggregate the data table identifiers of each synchronization data table to obtain the table-level customization rule of the distributed database cluster.

[0091] In the flow relationship table, exclude all second data entries related to the currently traversed distributed database cluster as the data flow target end, and the remaining data entries are the third data entries. The data tables recorded in the third data entries are the data tables that the current distributed database cluster needs to synchronize. These tables are the source tables in the data flow, and their data needs to be copied to other clusters.

[0092] The acquisition processor system aggregates the data table identifiers (such as table name, table structure, etc.) of all synchronized data tables to form table-level customization rules for the distributed database cluster. It should be noted that through screening and aggregation operations, the system can generate accurate table-level customization rules for each distributed database cluster, which facilitates subsequent data management and maintenance.

[0093] In this embodiment,

[0094] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. It should be noted that the execution subject of this embodiment can be any distributed database cluster in each distributed database cluster, and the distributed database cluster is deployed on a computing service device with data processing, network communication and program running functions. The following describes this embodiment with the distributed database cluster as the execution subject.

[0095] Based on this, an embodiment of the present application provides a database bidirectional synchronization method. In this embodiment, the database bidirectional synchronization method includes steps A10 to A20:

[0096] Step A10: if the distributed database cluster is a source-end database cluster, the source data table is updated according to the received external insert data change operation, the data change operation is written into the first archive log to obtain a second archive log, and the data change record corresponding to the data change operation is determined from the second archive log based on the preset table-level customization rule and the preset synchronization task, and the data change record is reported to the acquisition processor system, wherein the acquisition processor system determines the target data table having a synchronization relationship with the source data table based on the preset table-level distribution rule, and determines the database cluster where the target data table is located from each distributed database cluster, and sends the data change record to the database cluster;

[0097] If the distributed database cluster receives an externally inserted data change operation, the distributed database cluster is determined to be a source database cluster. The source database cluster updates the corresponding source data table according to the received data change operation to ensure the accuracy and consistency of the data, and records the data change operation in the first archive log to ensure the traceability and integrity of the data. Subsequently, according to the preset rules and format, the valid data change operation is screened out from the first archive log to generate a second archive log. The second archive log is an archive log specifically used for data synchronization, which contains all data change records that the source database cluster needs to send to the target database cluster. The source database cluster reports the screened data change records to the acquisition processor system.

[0098] The acquisition processor system determines the target data table having a synchronization relationship with the source data table based on the preset table-level distribution rules, determines the database cluster where the target data table is located from each distributed database cluster, and sends the data change record to the database cluster.

[0099] It is understandable that by recording data change operations in archive logs, the traceability and integrity of the data are ensured. The preset table-level customization rules and synchronization tasks make the data synchronization process more flexible and controllable, and can be adjusted and optimized according to actual needs, thereby improving the flexibility of data synchronization.

[0100] Step A20: if the distributed database cluster is the target database cluster, a data synchronization operation is performed on the target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and the data synchronization operation is written into the third archive log.

[0101] If the distributed database cluster is the target database cluster to receive the synchronized data, the target database cluster receives the second data change record from the acquisition processor system, and performs data synchronization operation on the corresponding target data table according to the received second data change record, and records the data synchronization operation in the third archive log. This is to record the process and results of data synchronization for subsequent data auditing and troubleshooting.

[0102] In one feasible implementation, the database bidirectional synchronization method further includes:

[0103] Step A30, parsing the preset synchronization task to obtain the cluster identity key and the first system identity information of each distributed database cluster;

[0104] The cluster identity key is a unique key used to verify and identify the identity of a distributed database cluster, for example, it may be an encrypted string that contains the identification information and security verification information of the cluster. The first system identity information refers to the system identity information of the cluster, which may include the name, version, serial number, etc. of the system.

[0105] By introducing the cluster identity key and the first system identity information, it can be ensured that only verified distributed database clusters can participate in the data synchronization process, thereby preventing unauthorized access and data leakage, and ensuring that each cluster has a unique identity, which helps to accurately identify and locate each cluster during the data synchronization process and avoid data synchronization errors or omissions.

[0106] Step A40, traverse each cluster identity key, decode the cluster identity key to obtain a cluster identifier and the second system identity information, and if the first system identity information is consistent with the second system identity information, save the cluster identifier.

[0107] Traverse the cluster identity key of each distributed database cluster and decode the cluster identity key to obtain the cluster identifier and second system identity information. The decoding process may involve decryption algorithms, checksum verification and other steps, which are not described here. Compare the decoded second system identity information with the first system identity information. If the two are consistent, it means that the cluster is legal and verified and can participate in the data synchronization process. At this time, save the cluster identifier of the cluster for subsequent use.

[0108] In one feasible implementation, step A10: determining the data change record corresponding to the data change operation from the second archive log based on the preset table-level customization rule and the preset synchronization task, and reporting the data change record to the acquisition processor system includes:

[0109] Step A101: parsing the second archive log through a preset synchronization task to obtain multiple parsed data change records, and filtering out data change records carrying non-current cluster identifiers from each parsed data change record to obtain candidate data change records, wherein the non-current cluster identifier is a cluster identifier of a database cluster other than the source database cluster in each distributed database cluster;

[0110] The source database cluster parses the second archive log through the preset synchronization task to obtain multiple parsed data change records, and filters out data change records carrying non-current cluster identifiers from each parsed data change record to obtain candidate data change records. It should be noted that the non-current cluster identifier is the cluster identifier of the database cluster other than the source database cluster in each distributed database cluster. Screening out data change records carrying non-current cluster identifiers can ensure that only data change operations related to the source database cluster are processed, reducing the amount of data to be processed subsequently, improving the efficiency of data synchronization, and only processing data change records related to the source database cluster, which helps to maintain the consistency and accuracy of data synchronization.

[0111] Step A102: based on a preset table-level customization rule, filter out data change records of non-synchronized data tables from candidate data change records to obtain a first data change record.

[0112] Based on the preset table-level customization rules, the first data change records of the data tables that need to be synchronized are further screened from the candidate data change records. The first data change records refer to those data change records that are both related to the source database cluster and meet the requirements of the table-level customization rules. Through the table-level customization rules, the data change records of the data tables that meet the table-level customization rules can be synchronized, which helps to maintain the accuracy of data synchronization and avoid unnecessary data transmission and processing.

[0113] In a feasible implementation manner, before step A20: performing a data synchronization operation on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system and writing the data synchronization operation into the third archive log, the step further includes:

[0114] Step A01, receiving the third system identity information, and decrypting the source cluster identity key set on the first connection session to obtain the source cluster identity identifier and the fourth system identity information, wherein the acquisition processor system creates a first connection session with the target database cluster, and sets the source cluster identity key on the first connection session;

[0115] The acquisition processor system creates a first connection session with the target database cluster and sets the source cluster identity key on the first connection session. The target database cluster receives the third system identity information and decrypts the source cluster identity key set on the first connection session to obtain the source cluster identity identifier and the fourth system identity information.

[0116] By decrypting the source cluster identity key, the acquisition processor system can verify the identity information of the connection session, thereby ensuring that only legitimate connection sessions can participate in the data synchronization process, effectively preventing unauthorized access and data leakage; the source cluster identity identifier and the fourth system identity information obtained after decryption can be used for subsequent connection session binding and verification processes, improving the accuracy and reliability of connection session management.

[0117] Step A02: If the third system identity information is consistent with the fourth system identity information, the source cluster identifier is bound to the first connection session to obtain a first identification session, and the source cluster identifier is bound to the second connection session to obtain a second identification session, wherein the second connection session is a connection session between the coordinator node and each data node in the target database cluster.

[0118] If the third system identity information is consistent with the fourth system identity information, the target database cluster binds the source cluster identifier to the first connection session to obtain a first identification session, and binds the source cluster identifier to the second connection session to obtain a second identification session. It should be noted that the second connection session is a connection session between the coordinator node and each data node in the target database cluster.

[0119] By binding the source cluster identifier to the connection session, it can be ensured that only legitimate connection sessions can participate in the data synchronization process, thereby ensuring the legitimacy and accuracy of data synchronization. In addition, through the management of identification sessions, data synchronization requests can be processed more efficiently, reducing unnecessary verification and inspection processes, and improving the efficiency of data synchronization.

[0120] In one feasible implementation, step A20: performing a data synchronization operation on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and writing the data synchronization operation into a third archive log, includes:

[0121] Step A201, receiving a data change operation statement based on a first identification session through a coordinator node, wherein the acquisition processor system parses a second data change record into a data change operation statement, and sends the statement to the coordinator node based on the first identification session;

[0122] The acquisition processor system parses the second data change record into a data change operation statement and sends it to the coordinator node based on the first identification session. The data change operation statement is received through the coordinator node based on the first identification session.

[0123] Step A202, sending the data change operation statement to the data node where the target data table is located based on the second identification session through the coordinator node, and executing the data change operation statement in the target data table;

[0124] The coordinator node uses the second identification session (which has also been verified and bound to the source cluster identifier in the previous step) to send the received data change operation statements to the data node where the target data table is located. After receiving these statements, the data node will perform corresponding data change operations in the target data table, such as inserting, updating or deleting data.

[0125] Step A203: Generate a log record based on the data change operation statement and the source cluster identifier, and store the log record in a third archive log.

[0126] The target database cluster generates log records based on the data change operation statement and the source cluster identifier. The log records contain detailed information about the data change operation, such as the operation type, operation time, operation object, etc., as well as the source cluster identifier, which is used to identify the source of the data change. After the log records are generated, they will be stored in the third archive log for subsequent data auditing and troubleshooting.

[0127] Compared with data synchronization processing based on databases, this embodiment uses data tables as basic units to control the synchronization flow of data changes among multiple clusters, avoiding synchronization of private data tables of database clusters during database synchronization, reducing synchronization of invalid data, thereby reducing performance loss and excess space usage during data synchronization and improving data synchronization efficiency.

[0128] At the same time, during each synchronization, table-level customized rules and synchronization tasks are used to screen out the data change records corresponding to the data change operations inserted outside the database cluster, and then synchronize the data change records, thereby realizing the synchronization of externally inserted data change operations, avoiding the distributed database cluster from synchronizing the data change operations received from other database clusters again, avoiding the loop conflict problem of the distributed database, and improving the security of data synchronization.

[0129] For example, to help understand the implementation process of the database bidirectional synchronization method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3A brief flowchart of a database bidirectional synchronization method is provided. Specifically: When the data change acquisition processor system of this embodiment is started, the cluster identity key will be obtained from each distributed database cluster. Then the table-level customization rules are refreshed in the cluster to establish a synchronization task. The synchronization task contains all the cluster identity keys of the system. The cluster will decrypt the cluster identity key to obtain the cluster identification of all clusters and the system identity information of the data change acquisition processor system. The cluster identification acts on the loop control, and the system identity information is used to identify whether the current synchronization task comes from a legitimate system. At this time, the synchronization initialization operation between multiple distributed database clusters is completed. External data insertion process: When the business client performs data insertion and other data change operations on the distributed cluster A, A returns a success message to the business client after writing the data changes to the data table and archive log. The synchronization task of distributed cluster A will parse the archive log in real time, and filter out the data change records carrying the B and C cluster identifiers by analyzing the cluster identifiers in the data changes (because if the data synchronized from clusters B and C to cluster A is not filtered, it will be synchronized out again as the data of cluster A, causing a loop). Then, by analyzing the data tables involved in the data changes, filter out the data changes in the table-level custom rule table that do not need to be synchronized, that is, filter the data change records of cluster identifiers B and C decrypted in the synchronization task, and filter the table changes that do not need to be synchronized in the table-level custom rule table. Finally, it is found that the data is a data change introduced from the outside and needs to be synchronized, and the data change is reported to the data change acquisition processor system. Data change synchronization process: The data change acquisition processor determines the target database for data changes according to the table-level distribution rules, first sets the cluster identity key connected to the target database, and then synchronizes the data changes. At this time, the distributed database cluster B will decrypt when receiving the cluster identity key A, bind the cluster identifier to the database connection after authentication, and receive the data change operation distributed by the data change acquisition processor to complete data synchronization. At the same time, when the synchronization task of distributed database cluster B parses the archive log of synchronized data changes, it first filters the data changes of cluster identifiers A and C. Since this data change is sent from cluster A to cluster B through the data change collection and processing system, and is not a data change introduced from outside the synchronization system, it needs to be filtered (if it is not filtered, it will be synchronized out again, causing a loop phenomenon), so as to achieve the purpose of loop control.

[0130] In this embodiment, for a distributed database cluster, each node contained in it is interconnected and independent of each other. The identity information of each data node configured with synchronization tasks in the distributed database cluster is different. The entire cluster needs to provide unique identity information as a whole, that is, the cluster identity key. Please refer to Figure 4The process of generating a cluster identity key can be: when the distributed database cluster is initialized and started for the first time, the global transaction manager node in the cluster will generate a cluster identifier, which is spliced ​​by the MAC address of the node and a random number to ensure the uniqueness of the identifier. The identifier is Base64 encoded to protect the security of the data to a certain extent. For example, for the unique identifier Identifier = (MACADDR<<16)|(random&OxFFFF), the MAC address MACADDDR = 00:50:56:a7:29:8e, the random number random = 22897942400000000010100000101011010100111001010011000111011110010 11100000-MACADDR 48bit-l random 16bit-l, the decimal number of the random number is 22613274093613792. Based on the above processing procedure, the cluster identifier is Mjl2MTMyNzQwOTM2MTM3OTI=. The global transaction manager node will record the cluster identifier in the control file under the node data directory, and will read it each time it is started, and will not be generated again. Other nodes in the cluster will actively obtain the identifier from the global transaction manager node and record it when they start. The global transaction manager node is selected to generate the cluster unique identifier because in a distributed database cluster, the coordinator node and data node can be composed of multiple nodes, while the global transaction manager node is unique. When the data change acquisition processor system obtains the cluster identity key from the global transaction manager node of the database cluster, it will send the system identity information to the cluster. The global transaction manager node outputs a cluster identity key that carries the cluster identifier and the data change acquisition processor system identity information by splicing the generated cluster unique identifier and the incoming system identity information, encrypting them with AES128 through the key files cipher and rand, and encoding them with Base64. It should be noted that the global transaction manager node uses the gs_guc generate command to generate the key files of the synchronization system: replsync.key.cipher ciphertext file and replsync.key.rand encryption factor file. After the distributed database cluster is initialized for the first time, the key file is sent to all nodes of all clusters in the synchronization system. For example, the cluster unique identifier is Mjl2MTMyNzQwOTM2MTM3OTI=, system identity information: A2347213, key files: replsync.key.cipher and replsync.key.rand, the identity key plain text is: MjI2MTMyNzQwOTM2MTM3OTI=|A2347213, the obtained cluster identity key is:

[0131] EX6yPi3+w2MRsOx2WAbeNZRrv1ncsKI70g04dqFAp0a0jxdFa2UtlQGlAyqrdmoK / wGErVTJFq85WvLbDWZpOW1AuLRXBrxiPGOsZUxTzIEz

[0132] In this embodiment, the cluster identity key is currently used in three stages:

[0133] (1) When configuring a synchronization task to the database, the identity information of the synchronization system (that is, the data change collection processor system) and the identity keys of all clusters will be carried. Figure 5 The process of saving the cluster identity can be to traverse the synchronization task cluster identity key, detect whether the traversal is completed, if the traversal is completed, determine the completion, if the traversal is not completed, decode the Base64 encoding of the current traversal cluster identity key, and then perform AES128 decryption to detect whether the decryption is successful. If the decryption is successful, perform system identity authentication. If the system identity authentication is successful, save the cluster identity for loop control. If the decryption fails or the parsed system identity information is incorrect, the authentication fails. (2) When the data change acquisition processor system establishes a streaming synchronization channel with the data node, the cluster identity key needs to be authenticated. Please refer to Figure 6 , the process of creating a streaming transmission channel is as follows: when starting the synchronization task, the data change acquisition processor system will create a connection session to each data node in the cluster, and then send the IDENTIFY_REPLSYNC command, carrying the identity key of the cluster to which the data node belongs and the identity information of the data change acquisition processor system. After decrypting the cluster identity key, the data node verifies the system identity information. If all are successful, it returns the verification success to the data change acquisition processor system. After that, the data change acquisition processor system will send the START_REPLICATION SLOT command to the data node to officially start streaming transmission. After the data node processes it, it returns success, and the streaming synchronization channel is officially connected. (3) When the data change acquisition processor system sets the cluster identity key on the database connection, the database will decrypt the key. If the decryption fails or the parsed system identity information is incorrect, the authentication fails. If it succeeds, the obtained cluster unique identifier will be bound to the database connection. Since the cluster identity key contains two parts, the cluster identifier and the system identity information, it is obtained and stored by the data change acquisition processor system, and there is no other way to obtain it. It is ensured that the identity key is only circulated within the synchronization system to prevent external acquisition and tampering.

[0134] The synchronous loop control in this embodiment relies on the collaboration of the distributed database cluster and the data change acquisition processor system. Figure 7, the loop control collaboration process is as follows: the distributed database cluster first needs to decrypt the configured cluster identity key to obtain the cluster identifier. The cluster identifier bound to the database connection of the coordinator node needs to be bound to each connection between the coordinator node and each data node in the session. Ensure that the data obtained by the data node has a corresponding cluster identifier and is recorded in the archive log. The cluster identifier can be bound to the database connection of the coordinator node through the session_bind_cluster_identifier function. When the coordinator and each data node establish a connection in this session, the data node actively queries the coordinator node for the cluster identifier bound to the current session, and completes the cluster identifier binding with the coordinator node through the session_bind_cluster_identifier function. At this time, when the data change is transmitted to the data node, the cluster identifier information in the session global variable will be obtained, and it will be stored together with the data change to form an archive log record. During the synchronization task, the data node will parse the archive log and analyze whether the cluster identifier in the data change record matches any cluster identifier obtained by decrypting the cluster identity key in the synchronization task. If the match is successful, the data change is filtered out because the data change is synchronized from other clusters to this cluster. If it is synchronized out again, it will cause a loop, so it needs to be filtered and not sent to the data change acquisition processor system. The data change acquisition processor system obtains the cluster identity key from each synchronization configured cluster and sends synchronization tasks to each cluster. The synchronization task contains all cluster identity keys within the synchronization system. When the data change report is received, the data change code stream is parsed into a configurable SQL statement. Set the synchronization source cluster identity key on the connection with the coordinator node and configure the SQL statement to the target database.

[0135] This embodiment implements a customized synchronization method based on data tables. In the actual use scenario of data synchronization, there are three ways to synchronize data tables: (1) synchronize to all database clusters in the system; (2) synchronize to a specified database cluster in the system; (3) do not participate in synchronization. Please refer to Figure 7As shown, table1 is a table that needs to be synchronized bidirectionally between two database clusters. Although table2 has the same definition, it is a data table unique to each cluster, just like table3 and table4, and does not need to be synchronized. If it is a synchronization scenario of multiple clusters, there will also be scenarios where some data tables only exist in a few clusters. If synchronization is performed based on the database as the basic unit, a large amount of invalid data will be synchronized, which greatly reduces the synchronization efficiency and also occupies the storage of the database cluster. This embodiment implements a customized synchronization method based on data tables as the basic unit. Users need to configure the source cluster and target cluster of the data table that needs to be concerned on the data change acquisition processor system. Taking the cluster data flow table in Table 1 below as an example (a total of 3 clusters), data table A is synchronized between clusters A and B, and is not synchronized to cluster C. Data table B is synchronized unidirectionally from cluster A to cluster C. If the target cluster of data table C is empty, it will not be synchronized. Other data tables not in the following table are considered to be synchronized to all clusters by default. It should be noted that users can customize the flow of data tables, and there is no need for synchronization. The data change acquisition processor system analyzes the table and then configures it to the cluster data node. The database performs efficient filtering in the early stage of decoding, thereby improving synchronization efficiency.

[0136] Table 1 Cluster data flow table

[0137] Data Sheet Source cluster Target cluster Data Table A Cluster A; Cluster B Cluster A; Cluster B Data Table B Cluster A Cluster C; Data Table C Cluster A

[0138] The data change collection processor system will parse the flow relationship of the data table, and configure the table that needs to be filtered by the cluster to the cluster during the configuration of the table-level custom rules (that is, collect the data tables that do not need to be synchronized). The process is as follows: identify the currently configured cluster number A; query all entries with cluster number A in the source cluster column; query all entries in the target cluster column that are empty or only contain cluster number A; the data table column is all the data tables that cluster A needs to filter, aggregate them into a string, and send them to the data node of cluster A as a table-level custom rule. After obtaining the table-level custom rules, the database cluster data node parses the string and inserts the table name into the hash table. Since each data change record needs to be filtered and judged, the query speed is faster. If the data change record needs to be synchronized, the table name is placed in front of the data change data in TLV encoding during the parsing process. The TLV encoding method can parse the table name more quickly and flexibly. For example, Table 2 below shows the data change of tableA. Each field in the table contains a unique identifier (TAG), a length value (LENGTH) and an actual value (VALUE). The first "1" in the table corresponds to the TAG (identifier) ​​of the first field, that is, TAG (1byte): 0x01. This TAG represents the "version number" field. The second "4" corresponds to the LENGTH (length value) of the first field, that is, LENGTH (1byte): 0x04. Here, "1byte" indicates the byte length occupied by LENGTH, and "0x04" is the hexadecimal value of LENGTH, indicating that the value of the version number field occupies 4 bytes. The third "1" is "VALUE (4bytes): 0x31", that is, it is the VALUE field immediately after LENGTH, and its TAG is 0x01 (version number field). The fourth "2" corresponds to the TAG (identifier) ​​of the second field, that is, TAG (1byte): 0x02, where "1byte" indicates the byte length occupied by TAG, and "0x02" is the hexadecimal value of TAG. This TAG represents the "table name" field. The fifth "6" corresponds to the LENGTH (length value) of the second field, that is, LENGTH (1byte): 0x06, where "1byte" indicates the byte length occupied by LENGTH, and "0x06" is the hexadecimal value of LENGTH, indicating that the value of the table name field occupies 6 bytes. "tableA" corresponds to the VALUE (value) of the second field, that is, VALUE (13bytes): 0x7461626c6541.

[0139] Table 2TableA data change table

[0140] 1 4 1 2 6 tableA Data changes

[0141] This embodiment proposes the concept of a distributed database cluster identity key. In the overall process of the synchronization system, the data change acquisition processor and the database cluster perform interactive authentication multiple times to ensure the security of the system, which can effectively prevent unauthorized clusters from being incorporated into the synchronization system due to the leakage of the identity. At the same time, the cluster identity obtained by decrypting the identity key is the core of the loop control. This embodiment proposes a loop control method that matches the synchronization architecture of the currently used distributed database cluster. This method proposes a binding method for the distributed cluster identity, and applies the cluster identity to all connections between the nodes of the cluster in the session, so that loop control can be performed in a distributed database, which is faster than the control speed of external tools, and does not require complex operations such as auxiliary tables. Based on ordinary loop control, this embodiment proposes a table-level customization method. This method can freely control the flow of data tables between multiple clusters in the synchronization system, and stop the synchronization of data tables.

[0142] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the database bidirectional synchronization method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0143] The present application also provides a database bidirectional synchronization device. The database bidirectional synchronization device provided by the present application adopts the database bidirectional synchronization method in the above embodiment, which can solve the technical problems of large performance loss, low security and low synchronization efficiency when performing data synchronization in a scenario with a large amount of data. Compared with the prior art, the beneficial effects of the database bidirectional synchronization device provided by the present application are the same as the beneficial effects of the database bidirectional synchronization method provided by the above embodiment, and the other technical features in the database bidirectional synchronization device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0144] The present application provides a database bidirectional synchronization device. It should be noted that the distributed database cluster and the data change acquisition processor system can be run on different devices or integrated on the same device. Therefore, the database bidirectional synchronization device provided by the present application may be a device that deploys a distributed database cluster and / or a data change acquisition processor system. Specifically, the database bidirectional synchronization device includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the database bidirectional synchronization method in the above embodiment.

[0145] Reference below Fig. 9, which shows a schematic diagram of the structure of a database bidirectional synchronization device suitable for implementing the embodiment of the present application. The database bidirectional synchronization device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The database bidirectional synchronization device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0146] like Fig. 9 As shown, the database two-way synchronization device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the database two-way synchronization device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the database two-way synchronization device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a database two-way synchronization device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0147] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0148] The database bidirectional synchronization device provided by the present application adopts the database bidirectional synchronization method in the above embodiment, which can solve the technical problems of large performance loss, low security and low synchronization efficiency when performing data synchronization in a scenario with a large amount of data. Compared with the prior art, the beneficial effects of the database bidirectional synchronization device provided by the present application are the same as the beneficial effects of the database bidirectional synchronization method provided by the above embodiment, and the other technical features in the database bidirectional synchronization device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0149] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0151] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the database bidirectional synchronization method in the above-mentioned embodiment.

[0152] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0153] The computer-readable storage medium may be included in the database bidirectional synchronization device; or may exist independently without being assembled into the database bidirectional synchronization device.

[0154] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the database two-way synchronization device, the database two-way synchronization device: (exclusive solution).

[0155] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0158] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned database bidirectional synchronization method, and can solve the technical problems of large performance loss, low security, and low synchronization efficiency when performing data synchronization in scenarios with large data volumes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the database bidirectional synchronization method provided by the above-mentioned embodiment, and will not be elaborated here.

[0159] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned database bidirectional synchronization method when executed by a processor.

[0160] The computer program product provided by the present application can solve the technical problems of large performance loss, low security, and low synchronization efficiency when performing data synchronization in scenarios with large data volumes. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the database bidirectional synchronization method provided by the above embodiment, and will not be elaborated here.

[0161] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A database bidirectional synchronization method, characterized in that: The database bidirectional synchronization method is applied to a data change acquisition processor system, and the database bidirectional synchronization method includes: Receiving data change records sent by source-end database clusters in each distributed database cluster, wherein the source-end database cluster receives an externally inserted data change operation, updates a source data table according to the data change operation, writes the data change operation into a first archive log to obtain a second archive log, determines a data change record corresponding to the data change operation from the second archive log based on a preset table-level customization rule and a preset synchronization task, and reports the data change record to the acquisition processor system; Based on a preset table-level distribution rule, a target data table having a synchronization relationship with the source data table is determined, and a target database cluster where the target data table is located is determined from each of the distributed database clusters, and the data change record is sent to the target database cluster, wherein the target database cluster performs a data synchronization operation on the target data table according to the data change record, and writes the data synchronization operation into a third archive log.

2. The database bidirectional synchronization method according to claim 1, characterized in that: The database bidirectional synchronization method comprises: Obtaining the cluster identity key of each of the distributed database clusters, generating a synchronization task based on each of the cluster identity keys, and sending the synchronization task to each of the distributed databases, respectively, wherein the distributed database determines the received synchronization task as a preset synchronization task; Generating respective table-level customization rules for each of the distributed database clusters, and sending each of the table-level customization rules to the corresponding distributed databases, wherein the distributed databases determine the received table-level customization rules as preset table-level customization rules; A streaming transmission channel with each of the distributed database clusters is created based on each of the cluster identity keys, wherein synchronization data is transmitted through the streaming transmission channel.

3. The database bidirectional synchronization method according to claim 2, characterized in that: The step of obtaining the cluster identity key of each of the distributed database clusters includes: The system identity information of the acquisition processor system is sent to each of the distributed database clusters respectively, wherein the global transaction manager in the distributed database cluster obtains the MAC address and the random number, and concatenates the MAC address and the random number to obtain an initial identifier, encodes the initial identifier to obtain a cluster identifier of the distributed database cluster, concatenates the acquisition processor system identity information and the cluster identifier to obtain concatenated information, encrypts the concatenated information through a preset key file to obtain a cluster identity key, and sends the cluster identity key to the acquisition processor system; Receive a cluster identity key sent by each of the distributed database clusters.

4. The database bidirectional synchronization method according to any one of claims 2 to 3, characterized in that: The step of generating table-level customization rules for each of the distributed database clusters comprises: Traversing each of the distributed database clusters, determining a first data entry with the distributed database cluster as a data flow source from a preset flow direction relationship table, wherein the flow direction relationship table is used to characterize the data flow direction relationship between each data table in each of the distributed database clusters; Determine, from the first data entry, a second data entry with the distributed database cluster as a data flow target end; The second data entry is filtered out from each data entry in the flow relationship table to obtain a third data entry, a data table in the third data entry is determined as a synchronization data table of the distributed database cluster, and data table identifiers of each synchronization data table are aggregated to obtain a table-level customization rule of the distributed database cluster.

5. A database bidirectional synchronization method, characterized in that: The database bidirectional synchronization method is applied to any one of the distributed database clusters, and the database bidirectional synchronization method includes: If the distributed database cluster is a source-end database cluster, the source data table is updated according to the received external insert data change operation, the data change operation is written into a first archive log to obtain a second archive log, and a data change record corresponding to the data change operation is determined from the second archive log based on a preset table-level customization rule and a preset synchronization task, and the data change record is reported to the acquisition processor system, wherein the acquisition processor system determines a target data table having a synchronization relationship with the source data table based on a preset table-level distribution rule, and determines a database cluster where the target data table is located from each of the distributed database clusters, and sends the data change record to the database cluster; If the distributed database cluster is a target database cluster, a data synchronization operation is performed on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and the data synchronization operation is written into a third archive log.

6. The database bidirectional synchronization method according to claim 5, characterized in that: The database bidirectional synchronization method also includes: Parsing the preset synchronization task to obtain the cluster identity key and the first system identity information of each of the distributed database clusters; Each of the cluster identity keys is traversed, and the cluster identity key is decoded to obtain a cluster identifier and second system identity information. If the first system identity information is consistent with the second system identity information, the cluster identifier is saved.

7. The database bidirectional synchronization method according to claim 6, characterized in that: The step of determining the data change record corresponding to the data change operation from the second archive log based on the preset table-level customization rule and the preset synchronization task, and reporting the data change record to the acquisition processor system includes: Parsing the second archive log by a preset synchronization task to obtain a plurality of parsed data change records, and filtering out data change records carrying non-current cluster identifiers from each of the parsed data change records to obtain candidate data change records, wherein the non-current cluster identifier is a cluster identifier of a database cluster other than the source database cluster in each of the distributed database clusters; Based on a preset table-level customization rule, data change records of a non-synchronized data table are screened out from the candidate data change records to obtain a first data change record.

8. The database bidirectional synchronization method according to any one of claims 5 to 7, characterized in that: Before the step of performing a data synchronization operation on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system and writing the data synchronization operation into a third archive log, the step further includes: receiving third system identity information, and decrypting the source cluster identity key set on the first connection session to obtain a source cluster identity identifier and fourth system identity information, wherein the acquisition processor system creates the first connection session with the target database cluster and sets the source cluster identity key on the first connection session; If the third system identity information is consistent with the fourth system identity information, the source cluster identifier is bound to the first connection session to obtain a first identification session, and the source cluster identifier is bound to the second connection session to obtain a second identification session, wherein the second connection session is a connection session between the coordinator node and each data node in the target database cluster.

9. The database bidirectional synchronization method according to claim 8, characterized in that: The step of performing a data synchronization operation on a target data table corresponding to the second data change record according to the second data change record received from the acquisition processor system, and writing the data synchronization operation into a third archive log includes: Receiving, through the coordinator node, a data change operation statement based on the first identification session, wherein the acquisition processor system parses the second data change record into the data change operation statement and sends it to the coordinator node based on the first identification session; Sending the data change operation statement to the data node where the target data table is located based on the second identification session through the coordinator node, and executing the data change operation statement in the target data table; A log record is generated based on the data change operation statement and the source cluster identifier, and the log record is stored in a third archive log.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the database bidirectional synchronization method according to any one of claims 1 to 9 are implemented.

Images