Method and system for bi-directional synchronization of data

Abstract

This application provides a method, system, and storage medium for bidirectional data synchronization, solving the "data loopback" problem that occurs during bidirectional data synchronization between two data sources, such as cloud and edge, in existing technologies. The method includes: acquiring first changed data from a first database; if the first changed data is business data, determining loopback identifier data based on the first changed data, the loopback identifier data including a loopback identifier; synchronizing the first changed data and the loopback identifier data to a second database; acquiring the logical log of the second database; parsing the logical log to obtain second changed data, the second changed data including the first changed data, the loopback identifier data, and third changed data, the third changed data being business data; filtering the second changed data based on the loopback identifier to remove the first changed data and the loopback identifier data, obtaining the third changed data; and sending the third changed data to a target data system.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a method and system for bidirectional data synchronization. Background Technology

[0002] During bidirectional data synchronization between different data sources, such as the cloud and the edge, a "data loop" phenomenon often occurs. For example, when data changes in the cloud, this changed data is synchronized to the edge, causing changes in the edge data. These changes are then synchronized back to the cloud, meaning the same data is repeatedly synchronized between the cloud and the edge, resulting in a "data loop." Solving the "data loop" problem has become one of the urgent technical issues that those skilled in the art need to address. Summary of the Invention

[0003] In view of this, embodiments of this application provide a bidirectional data synchronization method to solve the "data loop" problem that occurs in the process of bidirectional data synchronization between two data sources, such as the cloud and the edge, in the prior art.

[0004] This application provides a bidirectional data synchronization method for a bidirectional data synchronization engine. The method includes: acquiring first changed data from a first database; if the first changed data is business data, determining loopback identifier data based on the first changed data, the loopback identifier data including a loopback identifier; synchronizing the first changed data and the loopback identifier data to a second database; acquiring a logical log from the second database; parsing the logical log to obtain second changed data, the second changed data including the first changed data, the loopback identifier data, and third changed data, the third changed data being the business data; filtering the second changed data based on the loopback identifier to remove the first changed data and the loopback identifier data, obtaining the third changed data; and sending the third changed data to a target data system, the target data system being a message middleware or the first database.

[0005] A second aspect of this application provides a bidirectional data synchronization system, including a first database, a bidirectional data synchronization engine, and a second database. The bidirectional data synchronization engine acquires first changed data from the first database, the first changed data being business data. Based on the first changed data, it determines loopback identifier data, the loopback identifier data including a loopback identifier. It synchronizes the first changed data and the loopback identifier data to the second database. It acquires a logical log from the second database. It parses the logical log to obtain second changed data, the second changed data including the first changed data, the loopback identifier data, and third changed data, the third changed data being the business data. Based on the loopback identifier, it filters the second changed data to remove the first changed data and the loopback identifier data, obtaining the third changed data. Finally, it sends the third changed data to the second database.

[0006] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the bidirectional data synchronization method provided in any of the above embodiments.

[0007] According to the data bidirectional synchronization method, system, and storage medium provided in this application, first changed data is obtained from a first database. Loopback identifier data, including a loopback identifier, is added to the first changed data. The first changed data and the loopback identifier data are then synchronized to a second database. Next, second changed data is captured from the second database. Based on the loopback identifier, the second changed data is filtered to remove the first changed data and the loopback identifier data, resulting in business data, i.e., third changed data. Thus, the data bidirectional synchronization engine implements a bidirectional synchronization process of writing the first changed data to the second database and capturing the first changed data from the second database. After capturing the first changed data from the second database, the first changed data is discarded and no longer synchronized to the first database, thereby solving the "data loopback" problem. Simultaneously, this data bidirectional synchronization method supports multiple types of data sources, including any one of MySQL, PgSQL, Oracle, and Redis databases. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating the data flow during the operation of a bidirectional data synchronization engine provided in an embodiment of this application.

[0009] Figure 2 This is a flowchart illustrating the bidirectional data synchronization method provided in the first embodiment of this application.

[0010] Figure 3 A flowchart of a bidirectional data synchronization method provided in the second embodiment of this application.

[0011] Figure 4 for Figure 3 The diagram shows the principle of the bidirectional data synchronization method.

[0012] Figure 5 A flowchart of a bidirectional data synchronization method provided in the third embodiment of this application.

[0013] Figure 6 for Figure 5 The diagram shows the principle of the bidirectional data synchronization method.

[0014] Figure 7 This is a flowchart illustrating the bidirectional data synchronization method provided in the fourth embodiment of this application.

[0015] Figure 8a The diagram shown is a partial grouping process diagram of the data change table provided in the first embodiment of this application.

[0016] Figure 8b This is a schematic diagram of a partial grouping process for the data change table provided in the second embodiment of this application.

[0017] Figure 9 This is a schematic diagram of the architecture of a bidirectional data synchronization system provided in the first embodiment of this application.

[0018] Figure 10 for Figure 9 The diagram shows the data flow of the bidirectional data synchronization process in the bidirectional data synchronization system.

[0019] Figure 11 This is a schematic diagram of the architecture of a bidirectional data synchronization system provided in the second embodiment of this application.

[0020] Figure 12 This is a schematic diagram of the architecture of the data synchronization system provided in the third embodiment of this application.

[0021] Figure 13 This is a structural block diagram of a bidirectional data synchronization device provided in the first embodiment of this application.

[0022] Figure 14 This is a structural block diagram of a bidirectional data synchronization device provided in the second embodiment of this application. Detailed Implementation

[0023] In related technologies, many scenarios require bidirectional data synchronization between different data sources. For example, primary healthcare institutions often have poor network conditions, frequently experiencing network outages or weak connections. When these institutions access a cloud-based Hospital Information System (HIS), to ensure that they can switch to the primary healthcare system's HIS and continue their current business processes during network outages without the user's awareness, real-time bidirectional data synchronization between the primary healthcare system and the cloud-based HIS is necessary.

[0024] Currently, popular bidirectional data synchronization solutions in the industry include a combination of Canal and Otter components, and a Debezium-based solution. The Canal and Otter combination only supports data retrieval from MySQL databases and does not support data retrieval from other commonly used data sources such as PostgreSQL, Oracle, and Redis, thus limiting its application. While the Debezium-based solution supports a wider range of data sources, it is only suitable for data extraction-transform-load (ETL) processes. Many issues involved in bidirectional data synchronization lack complete solutions, such as data loopback, PostgreSQL database cluster master-slave switching, data splitting and synchronization, and on-demand synchronization.

[0025] The "data loop problem" refers to the issue of data being repeatedly synchronized between two data sources.

[0026] The "PostgreSQL database cluster master-slave failover issue" refers to the problem where Debezium, when fetching data from a PostgreSQL cluster, needs to create logical replication slots on the master node. These slots record information such as the start position of logical log synchronization. When a master-slave failover occurs in the PostgreSQL cluster, Debezium loses the information in the original logical replication slots when connecting to the new master node, resulting in data loss.

[0027] The "data splitting and synchronization problem" refers to the need to synchronize cloud data to different hospitals according to dimensions such as "organizational structure," requiring data to be split and synchronized. However, the Debezium solution cannot execute dynamic scripts to split and synchronize table data to different databases by attributes, nor does it have data stream orchestration capabilities.

[0028] The "on-demand synchronization problem" refers to the issue of synchronizing data between heterogeneous databases when the table names and field names of the source and target tables are not exactly the same. The Debezium solution cannot map table names to field names, and cannot meet the requirement of synchronizing only fields with the same names in the source and target tables. This requires the business side to write additional custom programs to implement this functionality, which is inconvenient for business users.

[0029] To address at least one of the aforementioned technical problems, embodiments of this application provide a data bidirectional synchronization method and system. The data bidirectional synchronization method is used in a data bidirectional synchronization engine, which can replace the combination of Canal and Otter, as well as debezium, in related technologies to achieve bidirectional synchronization between different data sources. The data bidirectional synchronization method includes: acquiring first changed data from a first database, the first changed data being business data; determining loopback identifier data based on the first changed data, the loopback identifier data including a loopback identifier; synchronizing the first changed data and the loopback identifier data to a second database; acquiring second changed data from the second database, the second changed data including the first changed data, the loopback identifier data, and third changed data, the third changed data being data generated by write operations of business data in the second database; filtering the second changed data based on the loopback identifier to remove the first changed data and the loopback identifier data, obtaining the third changed data. It can be seen that after acquiring the first changed data from the first database, the data bidirectional synchronization engine adds loopback identifier data, the loopback identifier data including a loopback identifier, and synchronizes the first changed data and the loopback identifier data to the second database. Next, the second changed data in the second database is captured. This second changed data is filtered based on the loopback identifier to remove the first changed data and the loopback identifier data, resulting in the business data, i.e., the third changed data. In this way, the bidirectional data synchronization engine implements a bidirectional synchronization process of writing the first changed data to the second database and capturing the first changed data from the second database. Furthermore, after capturing the first changed data from the second database, it discards the first changed data and no longer synchronizes it to the first database, thus solving the "data loopback" problem.

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Figure 1 This is a schematic diagram illustrating the data flow during the operation of a bidirectional data synchronization engine provided in an embodiment of this application. Figure 1 As shown, the bidirectional data synchronization engine 10 includes a core (change data capture core, abbreviated as cdc-core) module 11, a data capture engine 12, and a distribution interface module 13.

[0032] The cdc-core module 11 serves as the framework for the bidirectional data synchronization engine 10, housing the data capture engine 12 and the distribution interface module 13. Specifically, the cdc-core module 11 provides operations for creating, running, stopping, and restarting the data capture engine 12; defines the output data format of the data capture engine 12 as JavaScript Object Notation (JSON) and / or Redis command format; defines the distribution interface for JSON format data and / or Redis command format; and provides tools for parsing and processing JSON format data. Simultaneously, the cdc-core module 11 also defines an Application Programming Interface (API).

[0033] Data capture engine 12 is used to capture data change information in the database. Data capture engine 12 integrates the debezium data capture library, thus possessing data capture capabilities for various data sources. Data capture engine 12 includes at least one of the following: MySQL data capture engine, PgSQL data capture engine, Oracle data capture engine, and Redis data capture engine. Specifically, the MySQL data capture engine is used to capture data change information in the MySQL database; the PgSQL data capture engine is used to capture data change information in the PgSQL database; the Oracle data capture engine is used to capture data change information in the Oracle database; and the Redis data capture engine is used to capture data change information in the Redis database.

[0034] The distribution interface module 13 is used to write data change information captured by the data capture engine 12, and / or data change information processed by the JSON format data parsing and processing tool provided by the cdc-core module 11, into the database. The distribution interface module 13 includes at least one of the following: a Rocekt Message Queue (RocektMQ) distribution interface, a database distribution interface, and a Redis distribution interface. The RocektMQ distribution interface is a pure Java, distributed, queue-based open-source message middleware that supports transactional messages, sequential messages, batch messages, timed messages, and message backtracking. The data capture engine 12 distributes JSON format data to RocektMQ. At least one topic can be set in RocektMQ for use with JSON format data. When multiple topics are set in RocektMQ, each topic corresponds to a different table. Subsequently, other data systems can subscribe to topics in RocektMQ to obtain data from RocektMQ. The Database distribution interface is used to write JSON format data into the database. The Database distribution interface can parse JSON data into SQL statements. Simultaneously, it provides table mapping, field mapping within tables, and the output of the intersection of mapped fields. It can replace and filter field and table names based on configured mapping relationships, greatly facilitating business applications and meeting various synchronization scenarios. The Redis distribution interface can parse Redis commands into Jedis format data.

[0035] In one embodiment, such as Figure 1 As shown, the bidirectional data synchronization engine 10 also includes a Springboot assembly module 14 for integrating the open-source Springboot to enable the bidirectional data synchronization engine 10 to integrate Springboot functionality.

[0036] In one embodiment, such as Figure 1As shown, the bidirectional data synchronization engine 10 also includes a NiFi module for integrating the Apache NiFi platform, giving it powerful data flow management capabilities and integrating it into the NiFi ecosystem. The Apache NiFi platform is an easy-to-use, powerful, and reliable data fetching, processing, and distribution system. Designed for data flow, Apache NiFi supports highly configurable indicator graph data routing, transformation, and system mediation logic, and supports dynamic data fetching from various data sources. Because the bidirectional data synchronization engine 10 integrates with the NiFi ecosystem, it can use NiFi's ExecuteGroovyScript and RouteOnAttribute components to execute Groovy scripts for data processing and data distribution, thus enabling dynamic script execution and data flow orchestration capabilities.

[0037] In one embodiment, the bidirectional data synchronization engine 10 supports master-slave failover for the PostgreSQL cluster. Specifically, the bidirectional data synchronization engine 10 pre-stores the logical replication slot information of the PostgreSQL cluster, which is a cluster built on Patroni and HAProxy. When the PostgreSQL database undergoes a master-slave failover, Patroni replicates the logical replication slot information to the slave node. This ensures that the bidirectional data synchronization engine 10 obtains the WAL logs from the slave node, which is now the new master node.

[0038] This application also provides embodiments that... Figure 1 The diagram illustrates the execution process of bidirectional data synchronization by the bidirectional data synchronization engine. The following detailed explanation of the bidirectional data synchronization process is provided in conjunction with the accompanying diagram.

[0039] Figure 2 This is a flowchart illustrating the bidirectional data synchronization method provided in the first embodiment of this application. (In conjunction with...) Figure 1 and Figure 2 As shown, the bidirectional data synchronization method 200 includes:

[0040] Step S210: Obtain the first changed data from the first database. The first changed data is business data. Business data refers to data generated by the database based on business operations. For example, a business operation is to modify the value of a target variable. When this business operation is executed, the value of the target variable in the first database is modified. Accordingly, the first database generates a data change message for this modification operation, and this data change message is the business data.

[0041] See Figure 1The bidirectional data synchronization engine can retrieve the first changed data from the first database from the message middleware. This first changed data retrieved from the message middleware is guaranteed to be business data. The message middleware can be, for example, RocektMQ. The first database 20 can be any of the following: MySQL, PgSQL, Oracle, or Redis. When the first database 20 is a MySQL database, the MySQL data capture engine retrieves the first changed data from RocektMQ. When the first database 20 is a PgSQL database, the PgSQL data capture engine retrieves the first changed data from RocektMQ.

[0042] The bidirectional data synchronization engine can also obtain the first changed data from the first database. In this case, before step S210, the process includes: obtaining logical logs from the first database, parsing the logical logs to obtain the changed data, filtering the changed data based on the loopback identifier, and the remaining business data, i.e., the first changed data. The process of filtering the changed data based on the loopback identifier can be found in the relevant description of step S250, and will not be detailed here.

[0043] Step S220: Determine loopback identification data based on the first change data. The loopback identification data includes a loopback identifier.

[0044] For example, loopback identification data can be pre-stored in the bidirectional data synchronization engine. When the bidirectional data synchronization engine obtains the first changed data, it can directly read the loopback identification data. Alternatively, the loopback identification data can be generated based on the first changed data. When the bidirectional data synchronization engine obtains the first changed data, it can perform a series of processing operations on the first changed data to obtain the loopback identification data.

[0045] Step S230: Synchronize the first change data and loopback identifier data to the second database. After receiving the first change data and loopback identifier data, the second database parses the first change data and loopback identifier data to obtain the corresponding operation statements. Based on the operation statements, the second database is modified, and the first change data and loopback identifier data are generated based on the modifications.

[0046] Step S240: Obtain the logical log of the second database, parse the logical log to obtain the second change data, which includes the first change data, loopback identifier data and the third change data, which is business data.

[0047] Specifically, the second change data is obtained from the second database. The second change data obtained from the second database includes three types of data: the first type is business data, i.e., the third change data; the second type is loopback data, i.e., data synchronized from the first database, i.e., the first change data; and the third type is loopback identifier data.

[0048] Step S250: Filter the second change data based on the loopback identifier to remove the first change data and the loopback identifier data, and obtain the third change data.

[0049] The first changed data is the data synchronized from the first database to the second database. If the bidirectional data synchronization engine captures the first changed data from the second database and then synchronizes it back to the first database, it will cause the first changed data to be synchronized back and forth between the first and second databases, i.e., a data loop problem. Therefore, the first changed data needs to be discarded and no longer synchronized to the first database. Loopback indicator data is manually added data used to identify the first changed data, so it also needs to be discarded and does not need to be synchronized to the first database.

[0050] Step S260: Send the third change data to the target data system, which is a message middleware or the first database.

[0051] According to the bidirectional data synchronization method provided in this embodiment, after the bidirectional data synchronization engine obtains the first changed data from the first database, it adds loopback identifier data, which includes a loopback identifier. The first changed data and the loopback identifier data are then synchronized to the second database. Next, the second changed data from the second database is captured. Based on the loopback identifier, the second changed data is filtered to remove the first changed data and the loopback identifier data, resulting in the business data, i.e., the third changed data. In this way, the bidirectional data synchronization engine realizes the bidirectional synchronization process of writing the first changed data to the second database and capturing the first changed data from the second database. Furthermore, after capturing the first changed data from the second database, the first changed data is discarded and no longer synchronized to the first database, thus solving the "data loopback" problem. Meanwhile, this bidirectional data synchronization method supports multiple types of data sources, including any one of MySQL, PgSQL, Oracle, and Redis databases.

[0052] Figure 3 A flowchart of a bidirectional data synchronization method provided in the second embodiment of this application. Figure 4 for Figure 3 The diagram illustrates the principle of the bidirectional data synchronization method. In this embodiment, it is combined with... Figure 3 and Figure 4 As shown, step S220 is specifically executed as follows:

[0053] Step S320: If the format of the first changed data is a structured query statement, read the pre-stored loopback identifier data from the bidirectional data synchronization engine.

[0054] When the first database is any one of MySQL, PgSQL, or Oracle, the first changed data is a structured query statement. The bidirectional data synchronization engine pre-stores loopback identifiers and loopback identifier data. In this embodiment, as... Figure 4 As shown, the loopback identifier is a variable pre-stored in the second database. This variable, for example, is the target field `id` in the target table `cdc_tag` in the second database. The loopback identifier data includes a first structured query statement and a second structured query statement. The first structured query statement is used to set the variable to a first preset value. For example, the first structured query statement could be "update cdc_tag set tag=1 where id=1". The second structured query statement is used to set the variable to a second preset value. For example, the second structured query statement could be "update cdc_tag set tag=0 where id=1". In this case, when the first changed data is determined to be in JSON format, the pre-stored first and second structured query statements can be directly read as the loopback identifier data. The variable is like a switch; the first structured query statement is used to turn the switch on, and the second structured query statement is used to turn the switch off.

[0055] Step S230 is specifically executed as follows:

[0056] In step S330, the first structured query statement, the first changed data, and the second structured query statement are sequentially synchronized to the second database. In this case, the subsequent bidirectional data synchronization engine can sequentially obtain the first structured query statement, the first changed data, and the second structured query statement from the second database.

[0057] Step S250 is specifically executed as follows:

[0058] Step S350: Identify the first structured query statement based on the loopback identifier and the first preset value; identify the second structured query statement based on the loopback identifier and the second preset value; filter out the first structured query statement, the second structured query statement, and the first change data between the first structured query statement and the second structured query statement.

[0059] Specifically, such as Figure 4As shown, the loopback identifier and the first preset value are matched with the second changed data; if a match is found, the current structured query statement is determined to be the first structured query statement, and the filtering operation is initiated. The loopback identifier and the second preset value are then matched with the statement following the first current structured query statement; if a match is found, the second current structured query statement is determined to be the second structured query statement, and the filtering operation is deactivated. This discards the first structured query statement, the second structured query statement, and the first changed data between the first and second structured query statements, leaving only the third changed data.

[0060] The bidirectional data synchronization method provided in this embodiment is applicable to the bidirectional data synchronization process between two databases of at least one type, such as MySQL, PgSQL, and Oracle, and can avoid data loop problems.

[0061] Figure 5 A flowchart of a bidirectional data synchronization method provided in the third embodiment of this application. Figure 6 for Figure 5 The diagram illustrates the principle of the bidirectional data synchronization method. In this embodiment, it is combined with... Figure 5 and Figure 6 As shown, step S220 is specifically executed as follows:

[0062] Step S520: If the format of the first changed data is a Redis command, generate loopback identifier data based on the first changed data.

[0063] Specifically, a first predetermined command is executed on the first changed data to obtain a target auxiliary loopback identifier. The target auxiliary loopback identifier includes a loopback identifier and a target value, with the target value corresponding to the first data change information. The first predetermined command can be an encryption command used to encrypt the first changed data, and the obtained auxiliary loopback identifier is equivalent to a key. The target auxiliary loopback identifier is then assembled with a second predetermined command to form loopback identifier data, and the second predetermined command is used to perform a write operation on the target auxiliary loopback identifier.

[0064] For example, see Figure 6The first changed data is `set a1`. The first predefined command can be an MD5 command in the Redis database. After executing the MD5 command on `set a1`, the `circle-key-md5` is obtained, which is the target auxiliary loopback identifier. Here, `circle-key` is the loopback identifier, and `md5` is the target value, which corresponds to the first changed data. The second predefined command can be `setex()120 1`, which indicates that the data in parentheses will be cached for 2 minutes. This data is then combined with the target auxiliary loopback identifier to form the loopback identifier data: `setex circle-key-md5 120 1`. Subsequently, after the loopback identifier data is written to the second database, the target auxiliary loopback identifier can be cached in the second database for 2 minutes, after which it will be automatically deleted.

[0065] Step S250 is specifically executed as follows:

[0066] Step S551: Match the loopback identifier with the second changed data. Continuing from the previous example, this means matching the circle-key with the second changed data.

[0067] Step S552: If a match is found, it indicates that the current Redis command is a loopback identifier. In this case, the current Redis command is filtered out.

[0068] Match the circle-key with the second changed data. If a match is found, it means that the current Redis command is setex circle-key-md5 120 1. In this case, setex circle-key-md5 120 1 needs to be filtered out.

[0069] Step S553: ​​If no match is found, it indicates that the current Redis command may represent either the first or third changed data. In this case, execute the first pre-defined command on the current Redis command to obtain the auxiliary loopback identifier.

[0070] Continuing with the previous example, if the current Redis command is the first changed data, i.e., set a1, executing the first pre-defined command on the current Redis command will result in circle-key-md5, which is the same as the target auxiliary loop identifier already stored in the second database.

[0071] If the current Redis command modifies the data in the third step, such as `SET b1`, executing the first pre-defined command on top of the current Redis command will yield the circle-key-md5 hash. · circle-key-md5 ·The target auxiliary loopback identifier, i.e., circle-key-md5, is different from that already stored in the second database. Specifically, the target values ​​corresponding to circle-key are different; one is an md5 hash. · The other is MD5.

[0072] Step S554: Determine whether the second database stores an auxiliary loopback identifier. If it does, meaning the auxiliary loopback identifier is the same as the target auxiliary loopback identifier, it indicates that the current Redis command represents the first changed data. Filter out the current Redis command and obtain the third changed data. If it does not, it indicates that the current Redis command represents the third changed data, and retain it.

[0073] The bidirectional data synchronization method provided in this embodiment is applicable to the bidirectional data synchronization process between Redis databases, avoiding data loop problems.

[0074] Figure 7 This is a flowchart illustrating the bidirectional data synchronization method provided in the fourth embodiment of this application. (In conjunction with...) Figure 1 and Figure 7 As shown, in this embodiment, the first changed data in the first database is a data change table, see reference. Figure 8a The diagram shown is a partial grouping process of the data change table provided in the first embodiment of this application. As shown in Figure 8, the change data table includes multiple records, and each record includes a correspondence between primary key, table name, operation type, and data. In this case, the bidirectional data synchronization method 700 and... Figure 2 The difference in the bidirectional data synchronization method 200 shown is that, before step S220, it further includes:

[0075] Step S710: Divide multiple records into multiple groups. The multiple groups include a first type group and / or a second type group. The first type group includes multiple operation information records with the same primary key in the same table. The second type group includes single operation information records with different primary keys in the same table. The primary keys in the first type group and the second type group of the same table are different.

[0076] Specifically, see Figure 8a First, the multiple records 800 are divided into multiple intermediate groups 810 according to the table name. The records in the intermediate group 810 have the same table name.

[0077] Secondly, see Figure 8b This is a schematic diagram illustrating a portion of the grouping process of the data change table provided in the second embodiment of this application. For example... Figure 8bAs shown, multiple intermediate groups 810 are divided into multiple groups 820 based on the primary key, with multiple records in each group 820 having the same primary key. Specifically, for each intermediate group 810, the number of records corresponding to different primary keys in the intermediate group 810 is counted. Records corresponding to each primary key with a count greater than 1 are grouped into a first-type group. Records corresponding to all primary keys with a count of 1 are grouped into a second-type group.

[0078] Step S220 is specifically executed as follows:

[0079] Step S720 involves determining the loopback identifier data for each of the multiple groups, i.e., adding loopback identifier data for each group separately. The specific process can be found in the execution process of step S320, and will not be repeated here.

[0080] Step S230 is specifically executed as follows:

[0081] Step S730: Synchronize multiple groups carrying loopback identifier data in parallel to the second database.

[0082] Specifically, for the first type group, multiple records in the first type group are synchronized to the second database in a predetermined order. The predetermined order here refers to the order in which the multiple records are retrieved.

[0083] For the second type group, the multiple records in the second type group are divided into multiple classes according to the operation type, and the multiple classes are synchronized to the second database in parallel.

[0084] According to the grouping method of the first changed data in the bidirectional data transmission method provided in this embodiment, the accuracy of data during the parallel synchronization of multiple groups can be ensured, which greatly improves the data transmission efficiency compared with the conventional sequential synchronization method.

[0085] This application also provides a bidirectional data synchronization system, including the bidirectional data synchronization engine provided in any of the above embodiments. Figure 9 This is a schematic diagram of the architecture of a bidirectional data synchronization system provided in the first embodiment of this application. Figure 9As shown, the bidirectional data synchronization system 90 includes a first database 91, a first bidirectional data synchronization engine 92, a first message middleware 93, a second database 94, a second bidirectional data synchronization engine 95, and a second message middleware 96. The first database 91, the first bidirectional data synchronization engine 92, and the first message middleware 93 are deployed in a first data center, which can be a cloud data center. The second database 94, the second bidirectional data synchronization engine 95, and the second message middleware 96 are deployed in a second data center, which can be an edge data center. Either the first database 91 or the second database 94 can be a PostgreSQL database, a MySQL database, an Oracle database, a MongoDB database, or a Redis database.

[0086] like Figure 9 The bidirectional data synchronization process of the data bidirectional synchronization system shown includes the process of synchronizing data from the cloud to the edge and the process of synchronizing data from the edge to the cloud. Figure 10 for Figure 9 The diagram shows the data flow of the bidirectional data synchronization process in the bidirectional data synchronization system. Among them, Figure 10 Only one message middleware is shown, deployed in the cloud, which combines... Figure 9 The functions of the first message middleware 93 and the second message middleware 96 are combined. Figure 9 and Figure 10 As shown, taking a community hospital accessing a cloud-based HIS service as an example, when the community hospital's client requests the cloud-based HIS service, the relevant data from the cloud-based HIS service is stored in the first database 91 in the cloud, such as a PostgreSQL database. The mapping between the PostgreSQL database and the HIS service is pre-defined and stored in a configuration file. The first database 91 generates a first logical log based on the write operations of the relevant data from the cloud-based HIS service. Depending on the type of the first database, the first logical log may be any of the following: binlog log, WAL log, or logminer log. Figure 1 As shown.

[0087] First, the first bidirectional data synchronization engine 92 retrieves the first logical log from the first database 91, parses the first logical log, and obtains the first changed data. When the first logical log is any one of the binlog log, WAL log, or logminer log, the first changed data is JSON data.

[0088] The system determines whether the first changed data is business data, loopback data, or loopback identifier data based on the loopback identifier. If the first changed data is determined to be business data, it is sent to the first message middleware 93. If the first changed data is determined to be loopback data or loopback identifier data, it is discarded.

[0089] Secondly, if the first changed data is business data, the second data bidirectional synchronization engine 95 obtains the first changed data from the first message middleware 93. The first changed data is then divided into multiple groups; see [link to details regarding the grouping process] for more information. Figure 7 The illustrated embodiment involves parsing multiple groups to obtain SQL statements. Pre-stored loopback identifier data, including the first and second SQL statements, is read from the second bidirectional data synchronization engine 95. The first SQL statement is added to the beginning of each group, and the second SQL statement is added to the end of each group. Multiple groups carrying loopback identifier data are then synchronized in parallel to the second database 94. This completes the process of synchronizing cloud data to the edge.

[0090] Next, after synchronizing the first changed data and loopback identifier data, the second database 94 generates the first changed data and loopback identifier data again. Simultaneously, the second database 94 also generates new business data based on the client's business operations, which is recorded as the third changed data.

[0091] Next, the second data bidirectional synchronization engine 95 retrieves the second logical log from the second database 94, parses the second logical log, and obtains the second changed data. The second changed data includes the first changed data, loopback identifier data, and the third changed data. Based on the loopback identifier, the second changed data is filtered to discard the loopback identifier data and the first changed data, leaving the third changed data, i.e., the business data, which is then sent to the message middleware.

[0092] Next, the first data bidirectional synchronization engine 92 obtains the third changed data from the message middleware and divides the third changed data into multiple groups. See [link to specific grouping process] for details. Figure 7 The illustrated embodiment involves parsing multiple groups to obtain SQL statements. Pre-stored loopback identifier data, including the first and second SQL statements, is read from the first bidirectional data synchronization engine 95. The first SQL statement is added to the beginning of each group, and the second SQL statement is added to the end of each group. Multiple groups carrying loopback identifier data are then synchronized in parallel to the first database 91. This completes the process of synchronizing edge data to the cloud.

[0093] In one embodiment, if the community hospital network experiences a problem and cannot access cloud services, the client will automatically switch to access the HIS service on the edge to ensure high availability. Data will then be synchronized from the edge to the cloud, following the same steps as above, only the synchronization direction is different. However, due to the network anomaly, data will accumulate in the second message middleware 16 on the edge. When the hospital network is restored, the data will automatically synchronize to the cloud, and the client will automatically switch to access the cloud.

[0094] Figure 11 This is a schematic diagram of the architecture of a bidirectional data synchronization system provided in the second embodiment of this application. Figure 11 As shown, the bidirectional data synchronization system 110 includes a first database 111, a second database 112, a first bidirectional data synchronization engine 113, and a second bidirectional data synchronization engine 114. The first database 111 and the first bidirectional data synchronization engine 113 are located in a first data center, while the second database 112 and the second bidirectional data synchronization engine 114 are located in a second data center. The first and second data centers are connected via an intranet. Specifically, the first database 111 and the second bidirectional data synchronization engine 114 are communicatively connected, and the second database 112 and the first bidirectional data synchronization engine 113 are communicatively connected. The first database 111 and the second database 112 are of the same type, such as a Redis database or a MySQL database.

[0095] Taking the first database 111 and the second database 112 as Redis databases as an example, the data bidirectional synchronization process of the data bidirectional synchronization system 110 includes: the Redis data capture engine in the first data bidirectional synchronization engine 113 obtains the first logical log of the first database 111 and parses the first logical log into the first changed data. It then determines whether the first changed data is loopback data based on the loopback identifier. If the first changed data is loopback data or loopback identifier data, it is discarded. If the first changed data is business data, loopback identifier data is added to the first changed data, and the first changed data and loopback identifier data are sent to the Nifi queue. The Redis distribution interface sends the first changed data to the second database. This completes the data synchronization process from the first database to the second database.

[0096] The Redis data capture engine in the second bidirectional data synchronization engine 113 acquires the second logical log of the second database 112 and parses it into second changed data. Based on the loopback identifier, it determines that the second changed data includes loopback data and loopback identifier data, and discards the loopback data and loopback identifier data to avoid data loopback issues.

[0097] The data synchronization process from the second database to the first database is similar and will not be described in detail here.

[0098] Figure 12 This is a schematic diagram of the architecture of the data synchronization system provided in the third embodiment of this application. Figure 12 As shown, this embodiment utilizes a bidirectional data synchronization engine to achieve unidirectional data synchronization. Specifically, as... Figure 12 As shown, the data synchronization system 120 includes a first database 121, a second database 122, and a bidirectional data synchronization engine 123. The first database 121 can be either a PostgreSQL (Post-Relational Database) or Oracle database. The second database can be an ODS (Optical Data Store) database. The bidirectional data synchronization engine 123 captures the logical logs of the first database 121, parses the logs, and obtains a changed data table. The changed data table includes multiple records, each containing a primary key-table name-operation type-data mapping. These records are divided into multiple groups, including a first type group and / or a second type group. The first type group includes multiple operation records with the same primary key in the same table, while the second type group includes single operation records with different primary keys in the same table. The primary keys in the first type group and the second type group for the same table are different. After parsing the multiple groups into SQL statements in parallel, the statements are synchronized to the second database in parallel. This implements the data transmission and aggregation function of the bidirectional data synchronization engine 123.

[0099] For example, such as Figure 12 As shown, the first database is the municipal reporting platform database, which includes 18 first databases 121. The data in these 18 first databases 121 comes from 18 districts and counties, and each database 121 includes multiple data tables. The bidirectional data synchronization engine 123 captures the data from the 18 first databases 121, summarizes it, divides it into multiple groups, and sends these groups in parallel to the ODS database of the early warning platform. This facilitates data analysis for early warning applications and enables early warning of infectious diseases.

[0100] This application also provides a two-way data synchronization device. Figure 13 This is a structural block diagram of the bidirectional data synchronization device provided in the first embodiment of this application. Figure 13As shown, the bidirectional data synchronization device 130 includes an acquisition module 131, a determination module 132, a synchronization module 133, and a filtering module 134. The acquisition module 131 acquires first changed data from a first database. The determination module 132, if the first changed data is business data, determines loopback identifier data based on the first changed data; the loopback identifier data includes a loopback identifier. The synchronization module 133 synchronizes the first changed data and the loopback identifier data to a second database. The acquisition module 131 also acquires the logical log of the second database, parses the logical log to obtain second changed data, which includes the first changed data, the loopback identifier data, and third changed data, where the third changed data is business data. The filtering module 134 filters the second changed data based on the loopback identifier to remove the first changed data and the loopback identifier data, obtaining the third changed data. The synchronization module 133 also sends the third changed data to a target data system, which is a message middleware or the first database.

[0101] According to the bidirectional data synchronization device provided in this embodiment, after acquiring the first changed data from the first database, loopback identifier data is added. The loopback identifier data includes a loopback identifier. The first changed data and the loopback identifier data are then synchronized to the second database. Next, the second changed data from the second database is captured. Based on the loopback identifier, the second changed data is filtered to remove the first changed data and the loopback identifier data, resulting in the business data, i.e., the third changed data. Thus, the bidirectional data synchronization device realizes the bidirectional synchronization process of writing the first changed data to the second database and capturing the first changed data from the second database. Furthermore, after capturing the first changed data from the second database, the first changed data is discarded and no longer synchronized to the first database, thereby solving the "data loopback" problem. Simultaneously, this bidirectional data synchronization method supports multiple types of data sources, including any one of MySQL, PgSQL, Oracle, and Redis databases.

[0102] In one embodiment, the determining module 132 is specifically used to: read pre-stored loopback identifier data from the data bidirectional synchronization engine when the format of the first changed data is a structured query statement.

[0103] Further, in one embodiment, the loopback identifier is a variable pre-stored in the second database; the loopback identifier data includes a first structured query statement and a second structured query statement, wherein the first structured query statement is used to set the variable to a first preset value, and the second structured query statement is used to set the variable to a second preset value. In this case, the synchronization module 133 is specifically used to: sequentially synchronize the first structured query statement, the first changed data, and the second structured query statement to the second database.

[0104] Furthermore, in one embodiment, the filtering module 134 is specifically used to: identify a first structured query statement based on a loopback identifier and a first preset value; identify a second structured query statement based on a loopback identifier and a second preset value; and filter out the first structured query statement, the second structured query statement, and the first change data between the first structured query statement and the second structured query statement to obtain third change data.

[0105] The bidirectional data synchronization device provided in this embodiment is applicable to the bidirectional data synchronization process between two databases of at least one type, such as MySQL, PgSQL, and Oracle, and can avoid data loop problems.

[0106] In another embodiment, the determining module 132 is specifically used to: generate loopback identifier data based on the first changed data when the format of the first changed data is a Redis command.

[0107] For example, a first predetermined command is executed on the first changed data to obtain a target auxiliary loopback identifier, which includes a loopback identifier and a first target value; the target auxiliary loopback identifier and the second predetermined command are assembled into loopback identifier data, which is used to perform a write operation on the auxiliary loopback identifier.

[0108] In this case, in one embodiment, the filtering module 134 is specifically used to: match the loopback identifier with the second change data; if a match is found, filter out the current Redis command; if a match is not found, execute a first predetermined command on the current Redis command to obtain an auxiliary loopback identifier, the auxiliary loopback identifier including a loopback identifier and a second target value; if the first target value and the second target value are the same, filter out the current Redis command to obtain the third change data.

[0109] The bidirectional data synchronization device provided in this embodiment is suitable for the bidirectional data synchronization process between Redis databases, avoiding data loop problems.

[0110] Figure 14 This is a structural block diagram of a bidirectional data synchronization device provided in the second embodiment of this application. In this embodiment, the first changed data includes multiple records, and each record includes a correspondence between primary key, table name, operation type, and data. For example... Figure 14As shown, the difference between the bidirectional data synchronization device 140 and the bidirectional data synchronization device 130 is that the bidirectional data synchronization device 140 further includes a grouping module 141, which is used to divide multiple records into multiple groups. The multiple groups include a first type group and / or a second type group. The first type group includes multiple operation information of the same primary key in the same table, and the second type group includes single operation information of different primary keys in the same table. The primary keys in the first type group and the second type group corresponding to the same table are different.

[0111] In this case, the determination module 132 is specifically used to determine the loopback identifier data based on multiple groups respectively. The synchronization module 133 is specifically used to synchronize the multiple groups carrying the loopback identifier data to the second database in parallel.

[0112] In one embodiment, the grouping module 141 is specifically used for: First, dividing multiple records into multiple intermediate groups according to table names. For example, for each intermediate group, counting the number of records corresponding to different primary keys in the intermediate group; dividing the multiple records corresponding to each primary key with a quantity greater than 1 into a first type group; and dividing the records corresponding to all primary keys with a quantity equal to 1 into a second type group. Second, dividing the multiple intermediate groups into multiple groups according to the primary key.

[0113] In one embodiment, the synchronization module 133 is specifically used to synchronize multiple records in the first type group to the second database in a predetermined order; and / or to divide the multiple records in the second type group into multiple classes according to the operation type, and synchronize the multiple classes to the second database in parallel.

[0114] According to the grouping method of the first changed data in the bidirectional data transmission device provided in this embodiment, the accuracy of data during the parallel synchronization of multiple groups can be ensured, which greatly improves the data transmission efficiency compared with the conventional sequential synchronization method.

[0115] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the bidirectional data synchronization method provided in any of the above embodiments.

[0116] Computer-readable storage media can take the form of any combination of one or more readable media. A readable storage medium can be any of the following forms: electrical, magnetic, optical, electromagnetic, infrared, semiconductor, or a combination thereof. Examples of readable storage media include: hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0117] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0118] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0119] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0120] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for bidirectional data synchronization, characterized in that, For a bidirectional data synchronization engine; the method includes: Retrieve the first change data from the first database; When the first changed data is business data, loopback identification data is determined based on the first changed data. The loopback identification data includes a loopback identifier. The business data represents data generated by the database based on business operations. Synchronize the first change data and the loopback identifier data to the second database; Obtain the logical logs of the second database; The logical log is parsed to obtain second change data, which includes the first change data, the loopback identifier data, and the third change data, wherein the third change data is the business data. The second change data is filtered based on the loopback identifier to remove the first change data and the loopback identifier data, thereby obtaining the third change data; The third change data is sent to the target data system, which is a message middleware or the first database.

2. The data bidirectional synchronization method according to claim 1, characterized in that, The step of determining the loopback identifier data based on the first change data includes: When the format of the first changed data is a structured query statement, the pre-stored loopback identifier data is read from the bidirectional data synchronization engine.

3. The data bidirectional synchronization method according to claim 2, characterized in that, The loopback identifier is a variable pre-stored in the second database; the loopback identifier data includes a first structured query statement and a second structured query statement, wherein the first structured query statement is used to set the variable to a first preset value, and the second structured query statement is used to set the variable to a second preset value; The step of synchronizing the first change data and the loopback identifier data to the second database includes: The first structured query statement, the first changed data, and the second structured query statement are sequentially synchronized to the second database.

4. The data bidirectional synchronization method according to claim 3, characterized in that, The step of filtering the second change data based on the loopback identifier to remove the first change data and the loopback identifier data, and obtaining the third change data, includes: The first structured query statement is identified based on the loopback identifier and the first preset value; The second structured query statement is identified based on the loopback identifier and the second preset value; The third change data is obtained by filtering out the first structured query statement, the second structured query statement, and the first change data between the first structured query statement and the second structured query statement.

5. The data bidirectional synchronization method according to claim 1, characterized in that, The step of determining the loopback identifier data based on the first change data includes: When the format of the first changed data is a Redis command, the loopback identifier data is generated based on the first changed data.

6. The bidirectional data synchronization method according to claim 5, characterized in that, The step of generating the loopback identifier data based on the first change data includes: Execute a first predetermined command on the first changed data to obtain a target auxiliary loopback identifier, wherein the target auxiliary loopback identifier includes the loopback identifier and a first target value; The target auxiliary loopback identifier and the second predetermined command are assembled into the loopback identifier data, which is used to perform a write operation on the auxiliary loopback identifier.

7. The data bidirectional synchronization method according to claim 6, characterized in that, The step of filtering the second change data based on the loopback identifier to remove the first change data and the loopback identifier data, and obtaining the third change data, includes: Match the loopback identifier with the second change data; If a match is found, the current Redis command will be filtered out. If no match is found, the first predetermined command is executed on the current Redis command to obtain an auxiliary loopback identifier, which includes the loopback identifier and the second target value. If the first target value and the second target value are the same, the current Redis command is filtered out to obtain the third changed data.

8. The data bidirectional synchronization method according to claim 1, characterized in that, The first changed data includes multiple records, and each record includes a correspondence between primary key, table name, operation type, and data. Before determining the loopback identifier data based on the first change data, the method further includes: The multiple records are divided into multiple groups, including a first type group and / or a second type group. The first type group includes multiple operation information records with the same primary key in the same table, and the second type group includes single operation information records with different primary keys in the same table. The primary keys in the first type group and the second type group corresponding to the same table are different. The step of determining the loopback identifier data based on the first change data includes: The loopback identification data is determined based on the multiple groups respectively; The step of synchronizing the first change data and the loopback identifier data to the second database includes: The multiple groups carrying the loopback identifier data are synchronized in parallel to the second database.

9. The data bidirectional synchronization method according to claim 8, characterized in that, The step of dividing the multiple records into multiple groups includes: Divide the multiple records into multiple intermediate groups according to the table name; Divide the multiple intermediate groups into multiple groups based on the primary key.

10. The data bidirectional synchronization method according to claim 9, characterized in that, The step of dividing the plurality of intermediate groups into multiple groups based on the primary key includes: For each intermediate group, count the number of records corresponding to different primary keys in the intermediate group; Group the multiple records corresponding to each primary key with a quantity greater than 1 into a first-type group; All records corresponding to primary keys with a quantity of 1 are grouped into the second type group.

11. The data bidirectional synchronization method according to claim 8, characterized in that, The step of synchronizing the multiple groups carrying the loopback identifier data to the second database in parallel includes: For the first type group, multiple records in the first type group are synchronized to the second database in a predetermined order; and / or For the second type group, the multiple records in the second type group are divided into multiple classes according to the operation type, and the multiple classes are synchronized to the second database in parallel.

12. A bidirectional data synchronization system, characterized in that, Includes a first database, a two-way data synchronization engine, and a second database; The bidirectional data synchronization engine acquires first changed data from the first database, which is business data representing data generated by the database based on business operations; determines loopback identifier data based on the first changed data, which includes a loopback identifier; and synchronizes the first changed data and the loopback identifier data to the second database. Obtain the logical log of the second database; parse the logical log to obtain the second change data, which includes the first change data, the loopback identifier data, and the third change data, wherein the third change data is the business data; filter the second change data based on the loopback identifier to remove the first change data and the loopback identifier data, thereby obtaining the third change data; send the third change data to the second database.

13. The data bidirectional synchronization system according to claim 12, characterized in that, It also includes a message middleware; the bidirectional data synchronization engine obtains the first changed data from the message middleware and sends the third changed data to the second database through the message middleware.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the bidirectional data synchronization method as described in any one of claims 1 to 11.

Images