Database synchronization method and device, equipment, medium and program product

Abstract

The invention provides a database synchronization method which can be applied to the technical field of big data. The database synchronization method comprises the following steps: writing a data table in a first database into a message component by a data replication platform based on a first data log from the first database; extracting, by an incremental replication tool, a backup field in a first data log from the first database, the backup field comprising: pre-value data and post-value data; a data table in a first database in a consumption message component is written into a second database by an incremental replication tool under the condition that the post-value data is empty or under the condition that the previous-value data is equal to the post-value data and the post-value data is not empty; and after the data table in the first database is written into the second database by the incremental copying tool, modifying the post-value data. The invention further provides a database synchronization device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data, and in particular to a database synchronization method, apparatus, device, medium and program product. Background Art

[0002] Data synchronization refers to the synchronization of data between the source database and the target database. In this process, due to the difference in data distribution between the source database and the target database, data redistribution is also required after data migration. Data redistribution includes stock data redistribution and incremental data redistribution. Incremental data redistribution includes forward incremental redistribution and reverse incremental redistribution. After the stock data redistribution is completed, forward and reverse incremental data redistribution is enabled. In order to avoid business impact and control the scope of impact, applications are gradually switched to the target database according to the cluster dimension. At the same time point, there are businesses that read and write to the source and target databases.

[0003] For incremental data redistribution, incremental replication tools are usually configured to achieve incremental data redistribution between the source and target databases. However, relying solely on incremental replication tools will lead to problems such as large replication delays, non-scalability, and data loss. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data synchronization method, apparatus, device, medium and program product that reduce replication delay, improve scalability and avoid data loss.

[0005] According to a first aspect of the present disclosure, a database synchronization method is provided, the method being applied to a database bidirectional replication system, the database bidirectional replication system comprising a first database, a second database, a data replication platform, a message component and an incremental replication tool, the method comprising: the data replication platform writing a data table in the first database into the message component based on a first data log from the first database; the incremental replication tool extracting spare fields from the first data log from the first database, the spare fields comprising: previous value data and subsequent value data; the incremental replication tool consuming the data table in the first database in the message component and writing it into the second database when the subsequent value data is empty, or when the previous value data is equal to the subsequent value data and the subsequent value data is not empty; and the incremental replication tool modifying the subsequent value data after the data table in the first database is written into the second database.

[0006] According to an embodiment of the present disclosure, after the incremental replication tool extracts the spare field in the first data log from the first database, the method further includes: the incremental replication tool intercepts data to suspend replication when the post-value data is not empty and the previous value data is empty, or when the previous value data is not equal to the post-value data and the previous value data and the post-value data are not empty.

[0007] According to an embodiment of the present disclosure, the data replication platform writes a data table in the first database into a message component based on a first data log from the first database, including: parsing the first data log to obtain a table name and a primary key value; calculating a hash value based on the primary key value; determining a partition in the message component based on the hash value to obtain a second mapping relationship; and writing the data table into the message component according to a preset first mapping relationship and the second mapping relationship, wherein the preset first mapping relationship includes a one-to-one mapping relationship between the table name and the subject in the message component.

[0008] According to an embodiment of the present disclosure, the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the post-value data is empty, or when the pre-value data is equal to the post-value data and the post-value data is not empty, including: consuming the data table in the first database in the message component based on preset N threads, and writing it into the second database, wherein the threads correspond one-to-one to the partitions, and N is a positive integer.

[0009] According to an embodiment of the present disclosure, the method further includes: the data replication platform storing a global transaction identifier of the first database according to a first cycle.

[0010] According to an embodiment of the present disclosure, the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the post-value data is empty, or when the previous value data is equal to the post-value data and the post-value data is not empty, including: the incremental replication tool stores the data offset according to a preset second period during the data replication process.

[0011] A second aspect of the present disclosure provides a database synchronization device, which is applied to a database bidirectional replication system, wherein the database bidirectional replication system includes a first database, a second database, a data replication platform, a message component, and an incremental replication tool, wherein the device includes: a database acceptance module, for the data replication platform to write a data table in the first database into the message component based on a first data log from the first database; a spare field extraction module, for the incremental replication tool to extract spare fields from the first data log of the first database, wherein the spare fields include: previous value data and subsequent value data; a replication execution module, for the incremental replication tool to consume the data table in the first database in the message component and write it into the second database when the subsequent value data is empty or when the previous value data is equal to the subsequent value data and the subsequent value data is not empty; and a spare field modification module, for the incremental replication tool to modify the subsequent value data after the data table in the first database is written into the second database.

[0012] According to an embodiment of the present disclosure, the device also includes: an interception module, which is used for the incremental replication tool to intercept data to suspend replication when the post-value data is not empty and the previous value data is empty, or when the previous value data is not equal to the post-value data and the previous value data and the post-value data are not empty.

[0013] According to an embodiment of the present disclosure, the database acceptance module is specifically used to parse the first data log to obtain a table name and a primary key value; calculate a hash value based on the primary key value; determine the partition in the message component based on the hash value to obtain a second mapping relationship; and write the data table into the message component according to a preset first mapping relationship and the second mapping relationship, wherein the preset first mapping relationship includes a one-to-one mapping relationship between the table name and the subject in the message component.

[0014] According to an embodiment of the present disclosure, the replication execution module is specifically used to consume the data table in the first database in the message component based on preset N threads and write it into the second database, wherein the threads correspond one-to-one to the partitions and N is a positive integer.

[0015] According to an embodiment of the present disclosure, the device further includes: a first flag storage module, configured to store, by the data replication platform, a global transaction identifier of the first database according to a first cycle.

[0016] According to an embodiment of the present disclosure, the device further includes: a second flag storage module, which is used for the incremental replication tool to store data offsets according to a preset second period during the data replication process.

[0017] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0018] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0019] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0020] In the embodiments of the present disclosure, in order to reduce the problems of large replication delay, non-scalability and data loss during database synchronization, the embodiments of the present disclosure set up a database synchronization system, and the data replication platform accesses the database, and as a producer, puts the database data into the message component, and the incremental replication tool finds the previous value data and the next value data of the spare field to determine whether to consume the corresponding data to realize data replication to the target database. The embodiments of the present disclosure can achieve the following beneficial effects: 1. Data replication is realized by the data replication platform, message component and incremental replication tool, which can reduce replication delay and increase scalability; 2. The incremental tool identifies and modifies the spare field to ensure that each data synchronization through the incremental replication tool is recorded, which can avoid data loss and data reflux problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The application scenario diagram of the database synchronization method according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 2 A flowchart of a database synchronization method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A schematic diagram of an architecture of a database reconstruction method according to an embodiment of the present disclosure is shown;

[0025] Figure 4 The following schematically shows an architecture diagram of a method for redistributing stock data according to an embodiment of the present disclosure;

[0026] Figure 5 The schematic diagram shows an architecture diagram of a forward and reverse incremental data redistribution method according to an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of an architecture of forward incremental data replication according to an embodiment of the present disclosure is shown;

[0028] Figure 7 A schematic diagram of an architecture of reverse incremental data replication according to an embodiment of the present disclosure is shown;

[0029] Figure 8 The flowchart of a data replication platform processing data during reverse incremental replication according to an embodiment of the present disclosure is schematically shown. ;

[0030] Fig. 9 A flowchart for avoiding data loops according to an embodiment of the present disclosure is schematically shown;

[0031] Fig.10 A flowchart schematically shows a method for processing data by an incremental replication tool during forward incremental replication according to an embodiment of the present disclosure;

[0032] Fig.11 A full flow chart of a zero data loss solution according to an embodiment of the present disclosure is schematically shown;

[0033] Fig.12 A schematic diagram showing a structural block diagram of a database synchronization device according to an embodiment of the present disclosure; and

[0034] Fig.13 A block diagram of an electronic device suitable for implementing a database synchronization method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0036] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0037] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0038] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0039] Before the embodiments of the present disclosure are disclosed in detail, the key technical terms involved in the embodiments of the present disclosure are explained one by one, as follows:

[0040] Global Transaction Identifier (GTID): It indicates the number of a committed transaction and is a globally unique number. It is actually composed of a universal unique identifier and a transaction unique identifier. The universal unique identifier is a unique identifier of a MySQL instance. The transaction unique identifier represents the number of transactions that have been committed on the instance and increases monotonically as transactions are committed. The following is a specific form of a global transaction identifier (GTID), for example: 3E11FA47-71CA-11E1-9E33-C80AA9429562:23.

[0041] Sharding algorithm: Database sharding algorithm is a method for splitting and distributing data in a distributed database system. It divides the data in the database into multiple fragments (or shards), each of which is stored on a different server or node to achieve horizontal distribution and load balancing of data. Common database sharding algorithms: range sharding, hash sharding, list sharding, consistent hash sharding.

[0042] Unitization: refers to taking a self-contained set that can complete all business operations as the basic unit of deployment.

[0043] Applications use sharded data to route to a shard based on the sharding algorithm. Due to architecture transformation (unit transformation), the sharding algorithm needs to be changed. If the new algorithm is used for routing, the original data will not be accessible. Therefore, data migration is required to ensure that the records are distributed according to the new algorithm, that is, data redistribution. Figure 3 The following schematically shows an architecture diagram of a database reconstruction method according to an embodiment of the present disclosure. Figure 3As shown, data redistribution is to transfer the data in the source database to the target database according to different shard distributions. For example, there are M shards in the source database and N shards in the target database. In this process, data redistribution is required.

[0044] Data redistribution is divided into existing data redistribution and incremental data redistribution. Figure 4 The following schematically shows an architecture diagram of a method for redistributing stock data according to an embodiment of the present disclosure. Figure 4 As shown, the redistribution of existing data is to calculate the data in the shards in the source database through the component cluster, calculate to ensure the distribution of the data in the target database, and transfer the data to the target database. Figure 5 The following schematically shows an architecture diagram of a forward and reverse incremental data redistribution method according to an embodiment of the present disclosure. Figure 5 As shown, incremental data redistribution includes forward incremental data redistribution and reverse incremental data redistribution, wherein the forward direction is to redistribute the incremental data from the source library to the target library, and the reverse direction is to redistribute the incremental data from the target library to the source library.

[0045] The replication of incremental data between the source database and the target database is often implemented using an incremental replication tool. The incremental replication tool directly connects the source database and the target database. When incremental data is generated in either the source database or the target database, the incremental replication tool detects the incremental data and then implements incremental replication in the other database. For example: 1. The forward incremental replication tool monitors the BINLOG of the source database and copies the data to the target shard according to the new algorithm for sharding the target database; 2. The reverse incremental replication tool monitors the BINLOG of the target database and copies the data to the source shard according to the old algorithm for sharding the source database. In addition, in order to avoid the problem of data backflow (for example, there is incremental data in the source database, and the incremental data of the source database is synchronized to the target database, and there is a risk that the incremental data will flow back from the target database to the source database) in the prior art, a spare field value is usually set to indicate the flow direction of the incremental data to avoid the risk of loops caused by data backflow.

[0046] However, the following technical problems still exist in the prior art: 1. Large replication delay: The incremental replication tool runs on a single machine, reads data from BINLOG, parses the data, and writes it to the database. If the number of migration tables is large and the data is updated frequently, the system will be busy, resulting in serious replication delays; 2. Lack of scalability: If the number of migration tables is large and the data is updated frequently, it is impossible to increase the number of migration devices as needed to improve replication efficiency; 3. Data loss problem: Since the data backup field value indicates the flow direction of incremental data, once the data transmitted from the source database to the target database is updated in the target database, the data will not be synchronized to the source database.

[0047] In order to solve the technical problems existing in the prior art, an embodiment of the present disclosure provides a database synchronization method, which is applied to a database bidirectional replication system, wherein the database bidirectional replication system includes a first database, a second database, a data replication platform, a message component and an incremental replication tool, and the method includes: the data replication platform writes a data table in the first database into the message component based on a first data log from the first database; the incremental replication tool extracts a spare field in the first data log from the first database, and the spare field includes: previous value data and subsequent value data; the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the subsequent value data is empty, or when the previous value data is equal to the subsequent value data and the subsequent value data is not empty; and the incremental replication tool modifies the subsequent value data after the data table in the first database is written into the second database.

[0048] In the embodiments of the present disclosure, in order to reduce the problems of large replication delay, non-scalability and data loss during database synchronization, the embodiments of the present disclosure set up a database synchronization system, and the data replication platform accesses the database, and as a producer, puts the database data into the message component, and the incremental replication tool finds the previous value data and the next value data of the spare field to determine whether to consume the corresponding data to realize data replication to the target database. The embodiments of the present disclosure can achieve the following beneficial effects: 1. Data replication is realized by the data replication platform, message component and incremental replication tool, which can reduce replication delay and increase scalability; 2. The incremental tool identifies and modifies the spare field to ensure that each data synchronization through the incremental replication tool is recorded, which can avoid data loss and data reflux problems.

[0049] Figure 1 The application scenario diagram of the database synchronization method according to the embodiment of the present disclosure is schematically shown.

[0050] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0051] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0052] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0053] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0054] It should be noted that the database synchronization method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the database synchronization device provided in the embodiment of the present disclosure can generally be set in the server 105. The database synchronization method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the database synchronization device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0055] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0056] The following will be based on Figure 1 The scene described by Figure 2~Figure 11 The database synchronization method of the disclosed embodiment is described in detail.

[0057] Figure 2 The flowchart of the database synchronization method according to the embodiment of the present disclosure is schematically shown.

[0058] like Figure 2 As shown, the database synchronization method of this embodiment includes operations S210 to S250, and the database synchronization method can be executed by the server 105.

[0059] In operation S210, the data replication platform writes a data table in the first database into a message component based on a first data log from the first database.

[0060] It should be emphasized that the embodiments of the present disclosure only specifically handle the synchronization of the database during the incremental replication process.

[0061] The database synchronization method is applied to a database bidirectional replication system, which includes: a first database, a second database, a data replication platform, a message component and an incremental replication tool. Figure 6 The following schematically shows an architecture diagram of forward incremental data replication according to an embodiment of the present disclosure. Figure 7 The following schematically shows an architecture diagram of reverse incremental data replication according to an embodiment of the present disclosure. Figure 6 and Figure 7 ,exist Figure 6 and Figure 7 The system includes: source database (first database or second database), data replication platform, message component, incremental replication tool and target database (second database or first database). Figure 6 In this case, the source database does not directly interact with the incremental replication tool. The incremental replication tool receives information from the message component. Figure 7 In this case, the target database does not interact directly with the incremental replication tool, which receives information from the data replication platform via the message component.

[0062] Combined with 6, 1. The data replication platform obtains data from the log (or BinLog) of the source database, and writes the data to the topic (or topic) corresponding to the message component according to the table name; 2. The incremental replication tool is deployed in a cluster, and each incremental replication tool process is assigned a specific topic (or topic). After obtaining data from the message component, it writes the data to the target database according to the new algorithm; Combined with Figure 7 As shown in the figure, forward incremental data replication is from the source code library to the target library, and reverse incremental data replication is from the target library to the source library. The principle of reverse incremental data replication is the same as that of forward incremental data replication.

[0063] According to an embodiment of the present disclosure, the data replication platform writes a data table in the first database into a message component based on a first data log from the first database, including: parsing the first data log to obtain a table name and a primary key value; calculating a hash value based on the primary key value; determining a partition in the message component based on the hash value to obtain a second mapping relationship; and writing the data table into the message component according to a preset first mapping relationship and the second mapping relationship, wherein the preset first mapping relationship includes a one-to-one mapping relationship between the table name and the subject in the message component.

[0064] Among them, the first mapping relationship pre-associates each table in the first database with the subject in the message primary key. One subject is responsible for one table, and the table in the data replication platform and the subject in the message can be associated; the second mapping relationship is calculated during the incremental replication process, which is used to determine the relationship between the primary key and the partition, and realize the data transmission to each partition under a certain subject in the message component.

[0065] According to an embodiment of the present disclosure, the data replication platform stores the global transaction identifier of the first database according to a first cycle.

[0066] For example, the data replication platform stores the global transaction identifier (GTID) of the source database at intervals of 10 seconds. After the data replication platform is interrupted and restarted, data is replicated based on the GTID at this point, that is, transactions are restarted in the event of an interruption.

[0067] Figure 8 The following schematically shows a flow chart of a data replication platform processing data during reverse incremental replication according to an embodiment of the present disclosure. Figure 8 As shown in the figure, taking reverse incremental replication as an example, the data replication platform obtains data from the log (or BinLog) of the source library, parses the data, obtains the table name and primary key value. According to the table name, the corresponding message component topic (or topic) is found, and the hash value of the primary key value is calculated. The hash value is modulo the total number of topic (or topic) partitions (or partitions) to obtain the specific partition, and the data is written to the corresponding topic partition.

[0068] In operation S220, the incremental replication tool extracts spare fields from the first data log of the first database, where the spare fields include previous value data and next value data.

[0069] Among them, a corresponding spare field is configured for each table in the database, and the spare field is used to store a migration flag. The migration flag can represent the change operation of the data in the data table, for example, recording the addition, update, deletion, etc. of data in a certain database. The migration flag can also be used to represent the change method of the data in the table. For example, the change method of the data in the table comes from the incremental tool, or the change method in the table comes from other methods.

[0070] The previous value data refers to the identifier of the spare field before the corresponding data is changed, and the next value data refers to the identifier in the spare field after the corresponding data is changed. The data change operation and change method can be determined by comparing the previous value data and the next value data, so as to determine the update logic of the subsequent incremental replication tool.

[0071] It should be noted that the spare field design in the embodiment of the present disclosure is different from the commonly used solution in the industry. The commonly used solution in the industry is to store only one identifier in the spare field, which is used to record the writing direction of the data, such as forward from the source library to the target library and backward in the reverse direction. Fig. 9 A flowchart for avoiding data loops according to an embodiment of the present disclosure is schematically shown. Fig. 9 As shown in the figure, in order to avoid business data loops, the industry reserves spare fields for each table to store data replication direction flags. When the source database is copied to the target database, forward is written; when the target database is migrated to the source database, backward is written. When the incremental replication tool performs forward replication, it determines whether the spare field value is backward. If it is, it means that the data is written from the target database to the source database, and it does not need to be copied to the target database at this time; otherwise, it is written to the target database. When the incremental replication tool performs reverse replication, it determines whether the spare field value is forward. If it is, it means that the data is written from the source database to the target database, and it does not need to be copied to the source database at this time; otherwise, it is written to the source database. However, using only a single spare field will cause data loss problems. For example, if the forward incremental replication tool writes a row of data to the target database, if this row of data is updated in the target database, because the reverse incremental replication tool determines that the spare field value is forward, forward indicates the end of the incremental data, when the target database is updated, the spare field is still forward. When the data needs to be updated from the target database to the source database again, it will be intercepted. Although this method can avoid data backflow, it will also cause the data to be unable to be copied back to the source database. Therefore, the embodiment of the present disclosure abandons the single identification solution and ensures the scalability of data replication by setting up corresponding fields for storing previous value data and subsequent value data.

[0072] The data structure of BINLOG in the embodiment of the present disclosure is schematically shown below:

[0073] Add (insert):

[0074] op_type:"INSERT",after_value:{NODE_POSITION:null,NODE_NAME:"REGREG",ID:100014},table:"TEST_1",before_value:{}

[0075] update:

[0076] op_type:"UPDATE",after_value:{NODE_POSITION:"simon",NODE_NAME:"simon",ID:1},table:"TEST_1",before_value:{NODE_POSITION:"test",NODE_NAME:"test",ID:1}

[0077] Delete (del):

[0078] op_type:"DELETE",after_value:{},table:"TEST_1",before_value:{NODE_POSITION:"test",NODE_NAME:"test",ID:100012}

[0079] The field of the previous value is recorded as before_value, and the field of the next value is recorded as after_value.

[0080] In the new addition, the previous value data is: empty, and the subsequent value data is NODE_POSITION:null,NODE_NAME:"REGREG",ID:100014;

[0081] In the update, the previous value data is: NODE_POSITION:"test",NODE_NAME:"test",ID:1, and the subsequent value data is NODE_POSITION:"simon",NODE_NAME:"simon",ID:1;

[0082] In the deletion, the previous value data is: NODE_POSITION:"test",NODE_NAME:"test",ID:100012, and the subsequent value data is empty.

[0083] In operation S230, the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the after-value data is empty, or when the before-value data is equal to the after-value data and the after-value data is not empty.

[0084] It should be emphasized that after each synchronization of data through the incremental replication tool, the incremental replication tool will perform a write operation on the fields in the post-value data and write a flag serial number.

[0085] Specifically, if the post-value data is empty, it means that the data has not been copied by the incremental copy tool, so it can be incrementally copied to another database to complete the synchronization of data addition. If the post-value data is not empty and the previous value data is equal to the post-value data, it means that the data is updated in the current database (not written by the incremental copy tool), so it can be copied to another database to complete the synchronization of data update.

[0086] According to an embodiment of the present disclosure, the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the post-value data is empty, or when the pre-value data is equal to the post-value data and the post-value data is not empty, including: consuming the data table in the first database in the message component based on preset N threads, and writing it into the second database, wherein the threads correspond one-to-one to the partitions, and N is a positive integer.

[0087] According to an embodiment of the present disclosure, the incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the post-value data is empty, or when the previous value data is equal to the post-value data and the post-value data is not empty, including: the incremental replication tool stores the data offset according to a preset second period during the data replication process.

[0088] Specifically, for example, an offset is submitted at an interval of 3 seconds. The offset is used to record the offset in the message so that after the incremental replication tool is interrupted and restarted, data can be obtained starting from the specified offset in the message.

[0089] Fig.10 A flowchart of an incremental replication tool processing data during forward incremental replication according to an embodiment of the present disclosure is schematically shown.

[0090] like Fig.10 As shown, the incremental replication tool is deployed in a cluster. Each machine processes data concurrently. Each thread consumes data from a specified topic partition. After parsing the data, it calculates the number of shards based on the primary key value according to the new routing algorithm and writes the data into the specified shard.

[0091] In operation S240, after the incremental replication tool finishes writing the data table in the first database into the second database, the subsequent value data is modified.

[0092] The modified value data is used for subsequent incremental replication tools to execute verification logic thereon. The modification method can be manually defined and will not be described in detail here.

[0093] In operation S250, the incremental replication tool intercepts data to suspend replication when the subsequent value data is not empty and the previous value data is empty, or when the previous value data is not equal to the subsequent value data and the previous value data and the subsequent value data are not empty.

[0094] The operation S250 is performed after the operation S220.

[0095] Specifically, when the post-value data is not empty and the pre-value data is empty, it means that the data has been copied through the incremental copy tool, so there is no need to copy the data through the incremental copy tool. When the pre-value data is not equal to the post-value data and the pre-value data and the post-value data are not empty, it also means that the data has been copied through the incremental copy tool, so there is no need to copy the data through the incremental copy tool, thus avoiding data backflow and data loop.

[0096] Fig.11 The following schematically shows a full flow chart of a data zero loss solution according to an embodiment of the present disclosure. Fig.11 As shown, the details are as follows:

[0097] (1) Preparation: Configure a corresponding spare field for each migration table. The spare field is used to store the migration flag. The incremental replication tool modifies the migration flag during the migration and determines whether to copy to the target database based on the migration flag.

[0098] (2) Parse the data and obtain the before_value and after_value data. Among them:

[0099] ① New operation: before_value is empty, after_value indicates the value after adding.

[0100] ②Update operation: before_value is the value before update, and after_value is the value after update.

[0101] ③ Delete operation, before_value is the value before deletion, and after_value is empty.

[0102] (3) Get the value of the table's spare field from the before_value and after_value data. In the following text, before_value is used to represent the value of the table's spare field in the before_value data, and after_value is used to represent the value of the table's spare field in the after_value data. Among them:

[0103] ① After value is empty: This means that this row of data is the first time to pass through the incremental data replication tool and needs to be replicated. The sequence number (each table corresponds to a sequence number object) is incremented by 1 and the sequence number is written to the after_value standby field.

[0104] ② The latter value is not empty and the former value is empty: This means that this row of data is written by the incremental copy tool and is not copied.

[0105] ③ The after value is not empty and the before value is not empty: 1) The after value is equal to the before value: It means that the record is updated by the business (not written by the incremental tool) and needs to be copied. The sequence number is incremented by 1 and the sequence number is written into the after_value standby field; 2) The after value is not equal to the before value: It means that the record is written by the incremental copy tool and is not copied.

[0106] In the embodiments of the present disclosure, in order to reduce the problems of large replication delay, non-scalability and data loss during database synchronization, the embodiments of the present disclosure set up a database synchronization system, and the data replication platform accesses the database, and as a producer, puts the database data into the message component, and the incremental replication tool finds the previous value data and the next value data of the spare field to determine whether to consume the corresponding data to realize data replication to the target database. The embodiments of the present disclosure can achieve the following beneficial effects: 1. Data replication is realized by the data replication platform, message component and incremental replication tool, which can reduce replication delay and increase scalability; 2. The incremental tool identifies and modifies the spare field to ensure that each data synchronization through the incremental replication tool is recorded, which can avoid data loss and data reflux problems.

[0107] Based on the above database synchronization method, the present disclosure also provides a database synchronization device. Fig.12 The device is described in detail.

[0108] Fig.12 The structure block diagram of the database synchronization device according to the embodiment of the present disclosure is schematically shown.

[0109] like Fig.12 As shown, the database synchronization device 1200 of this embodiment includes a database receiving module 1210 , a spare field extraction module 1220 , a replication execution module 1230 and a spare field modification module 1240 .

[0110] The database receiving module 1210 is used for the data replication platform to write the data table in the first database into the message component based on the first data log from the first database. In one embodiment, the database receiving module 1210 can be used to perform the operation S210 described above, which will not be described in detail here.

[0111] The spare field extraction module 1220 is used to extract the spare field in the first data log from the first database by the incremental replication tool, and the spare field includes: previous value data and subsequent value data. In one embodiment, the spare field extraction module 1220 can be used to perform the operation S220 described above, which will not be repeated here.

[0112] The replication execution module 1230 is used for the incremental replication tool to consume the data table in the first database in the message component and write it into the second database when the after-value data is empty, or when the before-value data is equal to the after-value data and the after-value data is not empty. In one embodiment, the replication execution module 1230 can be used to perform the operation S230 described above, which will not be repeated here.

[0113] The spare field modification module 1240 is used to modify the post-value data after the incremental replication tool finishes writing the data table in the first database to the second database. In one embodiment, the spare field modification module 1240 can be used to perform the operation S240 described above, which will not be described in detail here.

[0114] In the embodiments of the present disclosure, in order to reduce the problems of large replication delay, non-scalability and data loss during database synchronization, the embodiments of the present disclosure set up a database synchronization system, and the data replication platform accesses the database, and as a producer, puts the database data into the message component, and the incremental replication tool finds the previous value data and the next value data of the spare field to determine whether to consume the corresponding data to realize data replication to the target database. The embodiments of the present disclosure can achieve the following beneficial effects: 1. Data replication is realized by the data replication platform, message component and incremental replication tool, which can reduce replication delay and increase scalability; 2. The incremental tool identifies and modifies the spare field to ensure that each data synchronization through the incremental replication tool is recorded, which can avoid data loss and data reflux problems.

[0115] According to an embodiment of the present disclosure, the device also includes: an interception module, which is used for the incremental replication tool to intercept data to suspend replication when the post-value data is not empty and the previous value data is empty, or when the previous value data is not equal to the post-value data and the previous value data and the post-value data are not empty.

[0116] According to an embodiment of the present disclosure, the database acceptance module is specifically used to parse the first data log to obtain a table name and a primary key value; calculate a hash value based on the primary key value; determine the partition in the message component based on the hash value to obtain a second mapping relationship; and write the data table into the message component according to a preset first mapping relationship and the second mapping relationship, wherein the preset first mapping relationship includes a one-to-one mapping relationship between the table name and the subject in the message component.

[0117] According to an embodiment of the present disclosure, the replication execution module is specifically used to consume the data table in the first database in the message component based on preset N threads and write it into the second database, wherein the threads correspond one-to-one to the partitions and N is a positive integer.

[0118] According to an embodiment of the present disclosure, the device further includes: a first flag storage module, configured to store, by the data replication platform, a global transaction identifier of the first database according to a first cycle.

[0119] According to an embodiment of the present disclosure, the device further includes: a second flag storage module, which is used for the incremental replication tool to store data offsets according to a preset second period during the data replication process.

[0120] According to an embodiment of the present disclosure, any multiple modules of the database acceptance module 1210, the spare field extraction module 1220, the copy execution module 1230, and the spare field modification module 1240 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the database acceptance module 1210, the spare field extraction module 1220, the copy execution module 1230, and the spare field modification module 1240 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the database acceptance module 1210, the spare field extraction module 1220, the copy execution module 1230 and the spare field modification module 1240 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0121] Fig.13 A block diagram of an electronic device suitable for implementing a database synchronization method according to an embodiment of the present disclosure is schematically shown.

[0122] like Fig.13As shown, the electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 to a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include an onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0123] In RAM 1303, various programs and data required for the operation of electronic device 1300 are stored. Processor 1301, ROM 1302 and RAM 1303 are connected to each other through bus 1304. Processor 1301 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1302 and / or RAM 1303. It should be noted that the program can also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0124] According to an embodiment of the present disclosure, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to the bus 1304. The electronic device 1300 may further include one or more of the following components connected to the input / output (I / O) interface 1305: an input portion 1306 including a keyboard, a mouse, etc.; an output portion 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1308 including a hard disk, etc.; and a communication portion 1309 including a network interface card such as a LAN card, a modem, etc. The communication portion 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output (I / O) interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage portion 1308 as needed.

[0125] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0126] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1302 and / or RAM 1303 described above and / or one or more memories other than ROM 1302 and RAM 1303.

[0127] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0128] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1301. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0129] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1309, and / or installed from the removable medium 1311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0130] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0131] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0132] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0134] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A database synchronization method, characterized in that: The method is applied to a database bidirectional replication system, which includes a first database, a second database, a data replication platform, a message component, and an incremental replication tool. The method comprises: The data replication platform writes a data table in the first database into a message component based on a first data log from the first database; Extracting, by the incremental replication tool, spare fields from the first data log of the first database, the spare fields comprising: previous value data and next value data; The incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the after-value data is empty or when the before-value data is equal to the after-value data and the after-value data is not empty; and After the data table in the first database is written into the second database by the incremental replication tool, the post-value data is modified.

2. The method according to claim 1, characterized in that After the incremental replication tool extracts the spare field in the first data log from the first database, the method further includes: The incremental replication tool intercepts data to suspend replication when the subsequent value data is not empty and the previous value data is empty, or when the previous value data is not equal to the subsequent value data and the previous value data and the subsequent value data are not empty.

3. The method according to claim 1 or 2, characterized in that: The step of writing the data table in the first database into the message component based on the first data log from the first database by the data replication platform includes: Parse the first data log to obtain a table name and a primary key value; Calculate a hash value based on the primary key value; Determine the partition in the message component based on the hash value to obtain a second mapping relationship; and The data table is written into the message component according to a preset first mapping relationship and the second mapping relationship, wherein the preset first mapping relationship includes a one-to-one mapping relationship between the table name and the subject in the message component.

4. The method according to claim 3, characterized in that The incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the after-value data is empty or when the before-value data is equal to the after-value data and the after-value data is not empty, including: The data table in the first database in the message component is consumed based on preset N threads and written into the second database, wherein the threads correspond to the partitions one by one and N is a positive integer.

5. The method according to claim 1, characterized in that The method further comprises: The data replication platform stores the global transaction identifier of the first database according to a first cycle.

6. The method according to claim 1, characterized in that The incremental replication tool consumes the data table in the first database in the message component and writes it into the second database when the after-value data is empty or when the before-value data is equal to the after-value data and the after-value data is not empty, including: The incremental replication tool stores the data offset according to a preset second period during the data replication process.

7. A database synchronization device, characterized in that: The device is applied to a database bidirectional replication system, which includes a first database, a second database, a data replication platform, a message component and an incremental replication tool. The device comprises: A database receiving module, configured to cause the data replication platform to write a data table in the first database into a message component based on a first data log from the first database; A spare field extraction module, used for extracting spare fields in the first data log from the first database by the incremental replication tool, wherein the spare fields include: previous value data and subsequent value data; a replication execution module, configured to consume the data table in the first database in the message component and write the data into the second database by the incremental replication tool when the after-value data is empty or when the before-value data is equal to the after-value data and the after-value data is not empty; and The spare field modification module is used to modify the post-value data after the incremental replication tool completes writing the data table in the first database into the second database.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images