Incremental data synchronization method and device for solving multi-partition factor sequential dependency

Abstract

The application discloses a kind of incremental data synchronization methods for solving multi-partition factor sequential dependence. Read the binary log of relational database in a time period, generate log message;In the table involved in the log message, the column in each table that is not allowed to appear duplicate value is called the partition factor of the table. The hash value of each partition factor of the row where the data change corresponding to each log message occurs is saved. Arrange partition for each log message, arrange partition when the associated log message is arranged in the same partition. The log message is saved in a global queue according to the order of the corresponding data change execution time, while recording the partition arranged for each log message;Using the watermark mechanism of Flink, log message with data change execution time earlier than the current watermark time is processed in turn. The application realizes the consistency of data synchronization in multi-partition scenario.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for incremental synchronization of a relational database (RDB). BACKGROUND

[0002] A relational database stores data in one or more tables, each of which is composed of rows and columns. A row in a table is called a record. MySQL is a widely used relational database management system.

[0003] A primary key is used to uniquely identify each row (i.e., each record) in a table.

[0004] The primary key is usually a column in a table, such as the "ID number" column in a personal information table or the "student number" column in a student information table. In this case, the value of the primary key column in each row in the table is unique and cannot be duplicated. The primary key is not allowed to contain NULL values.

[0005] The primary key can also be multiple columns in a table, which is called a composite primary key (also known as a joint primary key or a combined primary key). In this case, the combination of columns included in the composite primary key must be unique, even if each column is individually duplicated. Using a composite primary key helps ensure data integrity and avoid duplicate records.

[0006] A unique key is used to ensure that the value of a column or combination of columns in a table is unique in the table. In each row in the table, the corresponding column contains a unique value, but NULL values are allowed, and a column can contain multiple NULL values.

[0007] Binlog (Binary Log) is a log file that records data changes that occur in a MySQL database. Binlog records data change operations such as insert, update, delete, and database structure changes performed on the database. Binlog stores events in the form of events, and each Binlog event includes information about the execution time.

[0008] Database synchronization refers to copying data from one database to another and maintaining data consistency between them. Full synchronization and incremental synchronization are two common methods of database synchronization.

[0009] The existing incremental synchronization method of the relational database is based on the order of the execution time of each log event (for example, Binlog event) recorded by the binary log (for example, Binlog) of the relational database, and is divided into the following two kinds.

[0010] The first existing technology is a single-partition scheme. First, the binary log of the relational database in a certain period of time is read to generate log messages (for example, Binlog messages). Then, the log messages are sent to the receiving party in a global queue as a single topic and a single partition, and the receiving party database performs incremental data synchronization according to the order of receiving each log message.

[0011] The processing efficiency of the single-partition scheme is limited. In order to improve the processing efficiency, the second existing technology is a multi-partition scheme. First, the binary log of the relational database in a certain period of time is read to generate log messages. Then, the log messages are divided into multiple topics, each topic is divided into multiple partitions, and the log messages corresponding to the same record (i.e., the same row in the same table) are always placed in the same partition queue according to the Hash value of the primary key (referred to as the partition factor) of the data change recorded by each log message. The receiving party is sent by multiple partition queues in parallel, and the receiving party database performs incremental data synchronization according to the order of receiving each log message. The multi-partition scheme is prone to order dependency problems when the primary key or unique key data changes, resulting in data consistency errors. Please refer to the following two examples.

[0012] Example 1: Please refer to Table 1, Table M has 2 rows and 2 columns. The c1 column is the primary key field, and the c2 column is the unique key field.

[0013] c1 (primary key field) c2 (unique key field) r1 13110270002 55b919dca6dccd45abb28baab5873970 r2 98572960002 6a943c149365619cdd4970ec799b6498

[0014] Table 1: Data of Table M

[0015] The following operations are performed on Table M in sequence: first, delete r1 row. Then, delete r2 row. Finally, insert r3 row, the value of c1 field is 13110270002 (the same as the value of c1 field of r1 row), and the value of c2 field is 6a943c149365619cdd4970ec799b6498 (the same as the value of c2 field of r2 row). The log messages corresponding to the three operations are shown in Table 2.

[0016] log message 1 delete 13110270002 55b919dca6dccd45abb28baab5873970 log message 2 delete 98572960002 6a943c149365619cdd4970ec799b6498 log message 3 insert 13110270002 6a943c149365619cdd4970ec799b6498

[0017] Table 2: Log messages corresponding to data changes of Table M

[0018] If the receiver executes in the correct order of log message 1, log message 2, log message 3, there will be only r3 row in table M at the end.

[0019] If the existing multi-partition scheme is adopted and the hash partitioning is based on the primary key, the hash value of the primary key of the r1 row corresponding to the data change recorded by log message 1 is equal to the hash value of the primary key of the r3 row corresponding to the data change recorded by log message 3, log message 1 and log message 3 are dispatched to the same partition queue, and log message 2 is dispatched to other partition queues, at this time, the sending order of different partition queues cannot be controlled. If the receiver executes in the wrong order of log message 1, log message 3, log message 2, the insert operation corresponding to log message 3 will fail because of the unique key conflict (the value of the c2 field of the r2 row that should be deleted but has not been deleted is the same).

[0020] Example 2: Please refer to Table 3, table N has 2 rows and 8 columns. The pk1 column, the pk2 column, and the pk3 column constitute a composite primary key, the uk1 column, the uk2 column, and the uk3 column are all unique key fields, and the sk1 column and the sk2 column are business primary keys.

[0021]

[0022]

[0023] Table 3: Data of table N

[0024] The following operations are performed on table N in turn: first, the r1 row is deleted. Then, the r2 row is updated to modify the value of the pk3 field to 307. Then, the r2 row is updated to modify the value of the uk1 field to a3b4c5d7. Then, the r2 row is deleted. Finally, the r3 row is inserted, with the pk1 field having a value of 105, the pk2 field having a value of 206, the pk3 field having a value of 308, the uk1 field having a value of a3b4c5d7, the uk2 field having a value of e7f8g9h0, the uk3 field having a value of i1j2k3l4, the sk1 field having a value of s5, and the sk2 field having a value of s6. The log messages corresponding to the 5 operations are shown in Table 4.

[0025] log message 1 delete 101 202 303 a1b2c3d4 e5f6g7h8 i9j0k1l2 s1 s2 log message 2 update 104 205 307 a3b4c5d6 e7f8g9h0 i1j2k3l4 s3 s4 log message 3 update 104 205 307 a3b4c5d7 e7f8g9h0 i1j2k3l4 s3 s4 log message 4 delete 104 205 307 a3b4c5d7 e7f8g9h0 i1j2k3l4 s3 s4 log message 5 insert 105 206 308 a3b4c5d6 e7f8g9h0 i1j2k3l4 s5 s6

[0026] Table 4: Log messages corresponding to data changes of table N

[0027] If the receiver executes in the correct order of log message 1, log message 2, log message 3, log message 4, log message 5, there will be only r3 row in table N at the end.

[0028] If the existing multi-partition scheme is adopted and the hash partitioning is performed according to the primary key, because the composite primary key is composed of three fields, the partition corresponding to the row record is different if any field changes, log message 1 will be delivered to the queue of partition A, log message 2, log message 3, log message 4 will be delivered to the queue of partition B, and log message 5 will be delivered to the queue of partition C, at this time, the sending order of different partition queues cannot be controlled. If the log messages are sent in the order of C->A->B partition, the receiver will execute in the wrong order of log message 5, log message 1, log message 2, log message 3, log message 4, at this time, the insertion operation corresponding to log message 5 will fail due to unique key conflict (the same as the initial value of the c2 field of the r2 row that should be deleted but has not been deleted).

[0029] From the above two examples, it can be found that whether the primary key of a single field is used as the partition factor or the composite primary key of multiple fields is used as the partition factor, in the multi-partition incremental data synchronization scheme, the receiver database may be in an error situation of unique key conflict due to the delivery of log messages to different partitions. In fact, the partition factor can also use a unique key (single field or multiple field combination) or a business primary key (single field or multiple field combination), and the same problem will occur. In a real production environment, the design of the database and the table is more complex, the number and dependency relationship of the partition factors are more complex, and therefore the data disorder and conflict problems are more diverse, the order and dependency relationship of the log events of the log message records are easily disturbed, data conflicts are caused, and it is difficult to effectively handle.

[0030] At present, neither the Kafka-based distributed message queue scheme nor the Flink stream computing framework can fundamentally solve the order dependency problem of multiple partition factors. Apache Flink is a framework and distributed processing engine for stateful computation over unbounded and bounded data streams. Although frameworks such as Apache Flink provide a watermark mechanism for time control of out-of-order events, they lack support for complex dependency relationships of multiple partition factors. Therefore, it is both necessary and challenging to design a method that can solve the order dependency problem of any number of partition factors while ensuring the efficiency of incremental data synchronization. SUMMARY

[0031] The technical problem to be solved by the present application is to design an incremental data efficient synchronization method in a multi-partition scenario, to solve the order dependency problem between complex partition factors, and to guarantee data consistency.

[0032] To solve the above technical problems, the application provides a kind of incremental data synchronization method for solving multi-partition factor sequential dependence, comprising the following steps.Step S1: reading the binary log of relational database in a time period, generating log message;In the table involved in the log message, the column in each table that does not allow duplicate values is referred to as the partition factor of the table.Step S2: according to the order of corresponding data change execution time, the hash value of each partition factor of the row where each log message corresponding data change occurs is calculated in turn, and the name of the table where each log message corresponding data change occurs and the hash value of each partition factor of the row where each log message corresponding data change occurs are saved.Step S3: each log message is arranged in a partition according to the order of log message corresponding data change execution time, and the log messages associated with each other are arranged in the same partition when arranging the partition;If there is a sequential dependence relationship between some data changes, then there is an association between the log messages corresponding to these data changes.Step S4: save the log message in a global queue according to the order of corresponding data change execution time, and record the partition arranged for each log message;Using the watermark mechanism of Flink, log messages with data change execution time earlier than the current watermark time are processed in turn, and the log messages are sent to the receiver database by the arranged partition, and the receiver database executes the corresponding data change operation according to the received log message;The current watermark time refers to the trigger time of Flink's watermark mechanism, which indicates that the system can safely process events earlier than the trigger time.

[0033] Further, in step S1, the binary log of the relational database is stored in the form of events, and each log event includes the information of the recorded data change execution time;Each log message corresponds to a log event, and each log message also contains the information of the corresponding data change execution time.

[0034] Further, in step S1, each table has at least one partition factor;The partition factor of each table is the primary key field, unique key field, part or all of the business primary key field of the table.

[0035] Preferably, in step S3, a global version number is also generated for each log message when arranging the partition for each log message, and the global version number has the characteristics of unique increment.

[0036] Further, in step S3, it is specifically divided into the following three cases.

[0037] Case one: if the table where the data change corresponding to the log message K occurs is different from the table where the data change corresponding to the previously saved log message occurs, it is considered that the log message K does not exist with all the previous log messages Association, log message K is distributed to any partition.

[0038] Case two: if the table where the data change corresponding to the log message K occurs is the same as the table where the data change corresponding to the saved previous log message occurs, and the hash value of all partition factors of the row where the data change corresponding to the log message K occurs is different from the hash value of all partition factors of the row where the data change corresponding to all previous log messages occurs, it is considered that the log message K has no association with all previous log messages, and the log message is distributed to any partition.

[0039] Case three: if the table where the data change corresponding to the log message K occurs is the same as the table where the data change corresponding to the saved previous log message occurs, and the hash value of at least one partition factor of the row where the data change corresponding to the log message K occurs is the same as the hash value of at least one partition factor of the row where the data change corresponding to the previous log message J occurs, it is considered that the log message K has association with the previous log message J, and the log message K is distributed to the same partition of the previous log message J.

[0040] Further, the distributing the log message K to any partition includes distributing the log message K to a new partition.

[0041] Preferably, the case one further includes assigning a global version number to the log message K when arranging the partition for the log message K. The case two further includes assigning a global version number to the log message K when arranging the partition for the log message K. The case three further includes assigning a global version number to the log message K when arranging the partition for the log message K, and updating the global version numbers of the previous log message J and all log messages having association with the log message J to be the same as the global version number of the log message K.

[0042] Further, in the step S4, for the log messages not reaching the current watermark time, the registration timer temporarily stores these log messages first, and then triggers the processing of these log messages when the subsequent current watermark time advances to the corresponding time.

[0043] The application also provides a device for incremental data synchronization solving the sequential dependence of multi-partition factors, comprising a partition factor generating unit, a partition factor caching unit, a partition arranging unit and a sequential processing unit. The partition factor generating unit is used for reading the binary log of the relational database in a time period and generating log messages; in the table involved in the log messages, the column in which the repeated value is not allowed to appear in each table is referred to as the partition factor of the table. The partition factor caching unit is used for sequentially calculating the hash value of each partition factor of the row where the data change corresponding to each log message occurs in the order of the data change execution time corresponding to the log message, and saving the name of the table where the data change corresponding to each log message occurs and the hash value of each partition factor of the row where the data change corresponding to each log message occurs. The partition arranging unit is used for arranging the partition for each log message in the order of the data change execution time corresponding to the log message, and arranging the log messages that exist in association in the same partition when arranging the partition; if there is a sequential dependence relationship between some data changes, then there is an association between the log messages corresponding to the data changes. The sequential processing unit is used for saving the log messages in a global queue in the order of the data change execution time corresponding to the log messages, and recording the partition arranged for each log message at the same time; the log messages whose data change execution time is earlier than the current watermark time are sequentially processed by using the watermark mechanism of Flink, and the log messages are sent to the receiver database by the arranged partition, and the receiver database executes the corresponding data change operation according to the received log messages; the current watermark time refers to the trigger time of the watermark mechanism of Flink, which indicates that the system can safely process the events earlier than the trigger time.

[0044] The application has the following technical effects: by analyzing the dependence relationship between different data changes (i.e. the dependence relationship between multi-partition factors), it is determined whether there is an association between the log messages, the log messages that exist in association are arranged in the same partition, the log messages in the global queue that are sorted by time are sequentially processed by using the watermark mechanism of Flink, and the data synchronization consistency in the multi-partition scenario is realized. The application is suitable for complex partition scenarios, supports the composite primary key and composite unique key composed of multiple fields, and is suitable for complex data models. The application is suitable for incremental synchronization of large-scale data sets, is suitable for high-concurrency data processing, such as e-commerce, finance and other business scenarios with high requirements for data consistency, and is suitable for distributed real-time computing, such as user behavior log analysis, order flow synchronization and other scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0045] figure 1 is a flowchart of the method for incremental data synchronization solving the sequential dependence of multi-partition factors provided by the application.

[0046] figure 2Fig. 1 is a structural diagram of the incremental data synchronization device for solving the sequential dependence of multiple partition factors proposed in the present application.

[0047] Reference numerals in the figure: partition factor generating unit 1, partition factor caching unit 2, partition arranging unit 3, sequential processing unit 4. DETAILED DESCRIPTION

[0048] Reference numerals in the figure: partition factor generating unit 1, partition factor caching unit 2, partition arranging unit 3, sequential processing unit 4. figure 1 The incremental data synchronization method for solving the sequential dependence of multiple partition factors proposed in the present application includes the following steps.

[0049] Step S1: Read the binary log of the relational database in a time period to generate log messages. The binary log of the relational database stores in the form of events, and each log event includes the information of the recorded data change execution time. Each log message corresponds to a log event, and each log message also contains the information of the corresponding data change execution time. In the table involved in the log message, the column in each table that is not allowed to have duplicate values is called the partition factor of the table. Each table has at least one partition factor, and there can be multiple partition factors. The partition factor of each table is usually part or all of the primary key field, unique key field, business primary key field of the table.

[0050] In the second example above, it is assumed that the pk1 column, pk2 column, pk3 column, uk1 column, uk2 column, uk3 column of table N are not allowed to have duplicate values, and the sk1 column, sk2 column are allowed to have duplicate values, so the partition factors of table N are 6, and the 6 partition factors are respectively the value of the pk1 field, the value of the pk2 field, the value of the pk3 field, the value of the uk1 field, the value of the uk2 field, and the value of the uk3 field. As can be seen from Table 4, the values of the 6 partition factors of the first row are different from the values of the 6 partition factors of the second row.

[0051] Step S2: Calculate the hash value of each partition factor of the row where the data change corresponding to each log message occurs in sequence according to the order of the corresponding data change execution time of the log message, save the name of the table where the data change corresponding to each log message occurs or other information that can uniquely distinguish different tables, and also save the hash value of each partition factor of the row where the data change corresponding to each log message occurs.

[0052] Step S3: Arrange the partition for each log message according to the order of the data change execution time corresponding to the log message, and consider the dependence relationship between the data changes corresponding to different log messages when arranging the partition, which means arranging the log messages that have association in the same partition for processing. If there is a sequential dependence relationship between certain data changes, then there is association between the log messages corresponding to these data changes.

[0053] Preferably, this step also generates a global version number for each log message when arranging partition for each log message, and the global version number has the property of unique increment. For example, the global version number generated for each log message is one more than the global version number generated last time.

[0054] If the table where the data change corresponding to the log message K occurs is different from the table where the data change corresponding to the saved previous log message occurs, it is considered that the log message K has no association with all the previous log messages, and the log message K is distributed to any partition (including a newly added partition). Preferably, at this time, the log message K is also assigned a global version number.

[0055] If the table where the data change corresponding to the log message K occurs is the same as the table where the data change corresponding to the saved previous log message occurs, and the hash value of all partition factors of the row where the data change corresponding to the log message K occurs is different from the hash value of all partition factors of the row where the data change corresponding to all the previous log messages occurs, it is considered that the log message K has no association with all the previous log messages, and the log message K is distributed to any partition (including a newly added partition). Preferably, at this time, the log message K is also assigned a global version number.

[0056] If the table where the data change corresponding to the log message K occurs is the same as the table where the data change corresponding to the saved previous log message occurs, and the hash value of at least one partition factor of the row where the data change corresponding to the log message K occurs is the same as the hash value of at least one partition factor of the row where the data change corresponding to the previous log message J occurs, it is considered that the log message K has association with the previous log message J, and the log message K is distributed to the same partition as the previous log message J. Preferably, at this time, the log message K is also assigned a global version number, and the global version numbers of the previous log message J and all the log messages associated with the log message J are updated to be the same as the global version number of the log message K.

[0057] In the second example above, all the log messages correspond to data changes occurring in the same table.

[0058] First, the log message 1 is arranged for partition, because there is no saved hash value of each partition factor of the row where the data change corresponding to any log message occurs, so the log message 1 is distributed to any partition (including a newly added partition), and the log message 1 is assigned a global version number V1.

[0059] Subsequently, partition is arranged for log message 2. As known from the first 2 rows of Table 4, the hash values of all partition factors of r2 row where the data change corresponding to log message 2 occurs are different from the hash values of all partition factors of r1 row where the data change corresponding to the previous log message 1 occurs. Therefore, log message 2 is dispatched into any partition (including newly added partition) and global version number V2 is assigned to log message 2.

[0060] Subsequently, partition is arranged for log message 3. As known from the 2nd-3rd rows of Table 4, the hash values of 5 partition factors of r2 row where the data change corresponding to log message 3 occurs are the same as the hash values of 5 partition factors of r2 row where the data change corresponding to the previous log message 2 occurs. This indicates that log message 3 is associated with the previous log message 2. Therefore, log message 3 is dispatched into the same partition as the previous log message 2, global version number V3 is assigned to log message 3, and global version number of the previous associated log message 2 is updated from V2 to V3.

[0061] Subsequently, partition is arranged for log message 4. As known from the 3rd-4th rows of Table 4, the hash values of 6 partition factors of r2 row where the data change corresponding to log message 4 occurs are the same as the hash values of 6 partition factors of r2 row where the data change corresponding to the previous log message 3 occurs. This indicates that log message 4 is associated with the previous log message 3. If log message 4 is compared with the previous log message 2, the same conclusion can be obtained. As known that log message 3 is associated with the previous log message 2, it indicates that log message 4 is associated with the previous log message 3 and log message 2. Therefore, log message 4 is dispatched into the same partition as the previous log message 2 and log message 3, global version number V4 is assigned to log message 4, and global version number of the previous associated log message 2 and log message 3 is updated from V3 to V4.

[0062] Subsequently, the partition for log message 5 is arranged. As can be seen from the last two rows of Table 4, the r3 row where the data change corresponding to log message 5 occurs has the same hash value of the two partition factors as the r2 row where the data change corresponding to the previous log message 4 occurs, which indicates that log message 5 is associated with the previous log message 4. If log message 5 is compared with the previous log message 3, the same conclusion can be obtained. If log message 5 is compared with the previous log message 2, the same conclusion can be obtained. Since it is known that log message 4 is associated with the previous log message 2 and log message 3, it is indicated that log message 5 is associated with the previous log message 4, log message 3 and log message 2. Therefore, log message 5 is distributed to the same partition as the previous log message 2, log message 3 and log message 4, and the global version number V5 is assigned to log message 5, and the global version numbers of the previous associated log message 2, log message 3 and log message 4 are updated from V4 to V5.

[0063] At this point, the example two is processed according to the step S3 of the present application, and the partitions arranged for all log messages and the global version numbers set are shown in Table 5.

[0064]

[0065] Table 5: Result of processing example two according to step S3 of the present application

[0066] As can be seen from Table 5, log message 1 can be arranged in a certain partition, and the partition factors of log message 2, log message 3, log message 4 and log message 5 have a dependent relationship and must be arranged in the same partition in the original order.

[0067] Step S4: save the log messages in a global queue in order of the corresponding data change execution time, and record the partition arranged for each log message. Use the watermark mechanism of Flink to process the log messages with data change execution time earlier than the current watermark time in turn, send the log messages to the recipient database by the arranged partition, and the recipient database executes the corresponding data change operation according to the received log message. The current watermark time is the trigger time of the watermark mechanism of Flink, which indicates that the system can safely process the events earlier than the trigger time. For example: the current watermark time is 2024-11-22 12:00:00, at this time the system can process all log messages with data change execution time earlier than 2024-11-22 12:00:00. For log messages that do not reach the current watermark time (for example, data change execution time is 2024-11-22 12:00:05), the system will register a timer to temporarily store them, and trigger the processing of these log messages when the subsequent current watermark time advances to the corresponding time. The watermark mechanism avoids discarding or processing events too early, preventing data backlog or errors. The watermark mechanism combined with the global queue of the present application can ensure that out-of-order events are not processed too early, and will not be permanently backed up, thereby ensuring the consistency of the processing order of log messages in the same partition and between different partitions. Even in the presence of network delays or uneven data scenarios, the watermark mechanism can dynamically adjust the current watermark time to avoid deadlock or data congestion of the global queue.

[0068] Referring to figure 2 The incremental data synchronization device for solving the sequence dependence of the multi-partition factor provided in the present application includes a partition factor generation unit 1, a partition factor caching unit 2, an arranged partition unit 3, and a sequence processing unit 4. figure 2 The device shown corresponds to figure 1 The method shown.

[0069] The partition factor generation unit 1 is used to read the binary log of the relational database in a time period and generate log messages. In the table involved in the log message, the column in each table that is not allowed to have duplicate values is referred to as the partition factor of the table.

[0070] The partition factor caching unit 2 is used to calculate the hash value of each partition factor of the row where the data change corresponding to each log message occurs in order of the corresponding data change execution time of the log message, save the name of the table where the data change corresponding to each log message occurs or other information that can uniquely distinguish different tables, and save the hash value of each partition factor of the row where the data change corresponding to each log message occurs.

[0071] The arrangement partition unit 3 is used for arranging partitions for each log message in the order of the data change execution time corresponding to the log message, and arranging the log messages associated in the same partition when arranging the partitions.

[0072] The sequential processing unit 4 is used for saving the log messages in a global queue in the order of the data change execution time corresponding to the log messages, and recording the arranged partitions for each log message; using the watermark mechanism of Flink, sequentially processing the log messages with data change execution time earlier than the current watermark time, sending the log messages to the receiver database by the arranged partitions, and executing the corresponding data change operation according to the received log messages by the receiver database.

[0073] Compared with the prior art, the present application has the following beneficial effects.

[0074] Firstly, the present application can effectively solve the order dependency problem of the enhanced data synchronization scheme of the multi-partition factor. Even if multiple log messages are arranged in different partitions, the correct order between the multi-partition factors can be maintained, and the data conflict caused by the partition error can be avoided.

[0075] Secondly, the incremental data synchronization method based on the order of the execution time of each event recorded by the binary log of the existing relational database usually only partially considers the time factor, and focuses on the processing order in the single partition, and the time sorting and dependency management support across the partitions are weak. The present application combines the time factor with the partition factor dependency relationship, and ensures the strict distribution and processing of the events in the time order through the global queue and the watermark mechanism, and realizes the time consistency and dependency consistency across the partitions. Even in the scene with serious disorder or complex partition factors, data conflict can be effectively avoided, so that the processing effect of the data synchronization in the multi-partition distribution is significantly improved.

[0076] The above is only the preferred embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors, characterized in that, Includes the following steps; Step S1: Read the binary log of the relational database within a time period and generate log messages; in the tables involved in the log messages, the column in each table that does not allow duplicate values ​​is called the partition factor of that table; Step S2: Calculate the hash value of each partition factor of the row where the data change occurred for each log message in the order of the execution time of the corresponding data change, and save the name of the table where the data change occurred for each log message, as well as the hash value of each partition factor of the row where the data change occurred for each log message. Step S3: Assign partitions to each log message according to the order of the execution time of the data changes corresponding to the log messages. When assigning partitions, place related log messages in the same partition. If there is a sequential dependency between some data changes, then there is a relationship between the log messages corresponding to these data changes. Step S4: Store log messages in a global queue according to the order of their corresponding data change execution times, and record the partition assigned to each log message; using Flink's watermarking mechanism, process log messages whose data change execution times are earlier than the current watermark time in sequence, and have the assigned partition send the log message to the receiving database. The receiving database performs the corresponding data change operation based on the received log message; the current watermark time refers to the trigger time of Flink's watermarking mechanism, indicating that the system can safely process events earlier than that trigger time.

2. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 1, characterized in that, In step S1, the binary log of the relational database is stored in the form of events. Each log event includes information about the execution time of the recorded data change. Each log message corresponds to a log event, and each log message also includes information about the execution time of the corresponding data change.

3. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 1, characterized in that, step... In S1, each table has at least one partition factor; the partition factor of each table is part or all of the primary key field, unique key field, and business primary key field of that table.

4. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 1, characterized in that, In step S3, when partitioning each log message, a global version number is also generated for each log message. The global version number has the characteristic of being unique and incrementing.

5. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 1, characterized in that, Step S3 is specifically divided into the following three cases; Scenario 1: If the table in which the data change corresponding to log message K occurred is a different table from the table in which the data change corresponding to the previously saved log message occurred, then log message K is considered to be unrelated to all previous log messages, and log message K is distributed to any partition. Scenario 2: If the table in which the data change corresponding to log message K occurred is the same table as the table in which the data change corresponding to the previously saved log message occurred, and the hash values ​​of all partition factors of the row in which the data change corresponding to log message K occurred are not the same as the hash values ​​of all partition factors of the row in which the data change corresponding to all previous log messages occurred, then it is considered that log message K is not associated with any of the previous log messages, and the log message is distributed to any partition. Scenario 3: If the table in which the data change corresponding to log message K occurred is the same table as the table in which the data change corresponding to a previously saved log message occurred, and the hash value of at least one partition factor of the row in which the data change corresponding to log message K occurred is the same as the hash value of at least one partition factor of the row in which the data change corresponding to a previous log message J occurred, then log message K is considered to be related to the previous log message J, and log message K is distributed to the same partition of the previous log message J.

6. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 5, characterized in that, Distributing log message K to any partition includes distributing log message K to a newly added partition.

7. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 5, characterized in that, Case 1 also includes: when allocating partitions for log message K, a global version number is also assigned to log message K; Scenario 2 also includes: when allocating partitions for log message K, a global version number is also assigned to log message K; Case 3 also includes: when allocating a partition for log message K, a global version number is also assigned to log message K, and the global version number of the preceding log message J and all log messages associated with log message J is updated to be the same as the global version number of log message K.

8. The incremental data synchronization method for resolving the sequential dependency of multiple partitioning factors according to claim 1, characterized in that, In step S4, for log messages that have not reached the current watermark time, a timer is registered to temporarily store these log messages until the current watermark time advances to the corresponding time, at which point the processing of these log messages is triggered.

9. An incremental data synchronization device for resolving sequential dependencies of multiple partitioning factors, characterized in that, It includes a partition factor generation unit, a partition factor caching unit, a partition arrangement unit, and a sequential processing unit; The partition factor generation unit is used to read the binary log of the relational database within a time period and generate log messages; in the tables involved in the log messages, the column in each table that does not allow duplicate values ​​is called the partition factor of that table. The partition factor caching unit is used to calculate the hash value of each partition factor of the row in which the data change occurred for each log message in the order of the execution time of the corresponding data change, and to save the name of the table in which the data change occurred for each log message, as well as the hash value of each partition factor of the row in which the data change occurred for each log message. The partitioning unit is used to partition each log message according to the order of the execution time of the data changes corresponding to the log messages. When partitioning, related log messages are placed in the same partition. If there is a sequential dependency between some data changes, then there is a relationship between the log messages corresponding to these data changes. The sequential processing unit stores log messages in a global queue according to the order of their corresponding data change execution times, and records the partition assigned to each log message. Using Flink's watermarking mechanism, log messages with data change execution times earlier than the current watermark time are processed sequentially. The assigned partition then sends the log message to the receiving database, which performs the corresponding data change operation based on the received log message. The current watermark time refers to the trigger time of Flink's watermarking mechanism, indicating that the system can safely process events earlier than that trigger time.

Images