Metadata change synchronization method and system, data synchronization method and storage medium

Abstract

The invention discloses a metadata change synchronization method and system, a data synchronization method and a storage medium, and the method comprises the steps: obtaining a pre-write log of a to-be-synchronized source end, and obtaining a modification operation event for source end data in the pre-write log in real time; analyzing a modification operation object and corresponding identification information from each modification operation event; for each modification operation event, executing the following operations: executing a corresponding modification operation based on a first preset rule under the condition of confirming that an object of the current modification operation is metadata, and generating a corresponding unsubmitted metadata snapshot after the modification operation is completed; under the condition that it is confirmed that the object of the current modification operation is data, corresponding modification operation is executed based on a second preset rule; the problem that in the prior art, due to the fact that metadata changes and data changes exist in the same transaction, the consistency of the transaction is damaged is solved, and the rollback operation of the transaction can be supported.

Description

Technical Field

[0001] The present invention relates to the technical field of database synchronization storage, and in particular to a metadata change synchronization method, system, data synchronization method and storage medium. Background Art

[0002] With the development of science and technology, software systems in all walks of life today cannot do without databases, but there is currently no database that can meet all needs. Therefore, there will be multiple database systems in the business, and metadata and data need to be synchronized in each database system.

[0003] A transaction is a sequence of operations to access a database. The database application system completes access to the database through a transaction set. The correct execution of a transaction causes the database to be converted from one state to another. A transaction is to package multiple database operations into an "indivisible" whole for execution. At the same time, after these operations are executed, the data must be reliable and consistent. In order to perfectly solve the problem of reliability and consistency, transactions must comply with the ACID principle established by ISO / IEC. Among them, ACID is the abbreviation of Atomicity, Consistency, Isolation, and Durability.

[0004] The existing technology has the following problems in synchronizing metadata changes:

[0005] If metadata changes and data changes are in one transaction, and the metadata changes and data changes in the transaction are committed separately, the consistency of the transaction is destroyed, and the rollback operation may not be supported. Summary of the invention

[0006] The present invention provides a metadata change synchronization method, system, data synchronization method and storage medium, aiming to effectively solve the technical problems existing in metadata change synchronization in the prior art.

[0007] According to one aspect of the present invention, the present invention provides a metadata change synchronization method, the method comprising:

[0008] Obtain the write-ahead log of the source end to be synchronized, and obtain the modification operation events for the source end data in the write-ahead log in real time;

[0009] Parsing the object of the modification operation and the corresponding identification information from each modification operation event;

[0010] For each modification operation event, perform the following operations:

[0011] When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated;

[0012] When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule;

[0013] Generate a statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statement to be synchronized to the target end.

[0014] Further, for each modification operation event, when it is confirmed that the object of the current modification operation is metadata, executing the corresponding modification operation based on the first preset rule includes:

[0015] Matching the metadata changes with uncommitted transactions according to the identification information,

[0016] If no corresponding uncommitted transaction is matched, a new uncommitted transaction is created, and the corresponding modification operation is performed based on the closest committed metadata snapshot;

[0017] If a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot.

[0018] Further, for each modification operation event, when it is confirmed that the object of the current modification operation is data, executing the corresponding modification operation based on the second preset rule includes:

[0019] Matching the data changes with uncommitted transactions according to the identification information,

[0020] If no corresponding uncommitted transaction is matched, the corresponding modification operation is performed based on the closest committed metadata snapshot;

[0021] If a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot.

[0022] Further, the generating of the statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs includes:

[0023] When it is confirmed that the object of the current modification operation is metadata, a metadata modification operation statement corresponding to the target end is generated;

[0024] When it is confirmed that the object of the current modification operation is data, a data change operation statement corresponding to the target end is generated.

[0025] Furthermore, the method further includes: synchronizing the metadata change operation statement and the data change operation statement belonging to the same transaction to the target end.

[0026] Furthermore, the identification information includes one or a combination of an event ID and a transaction ID.

[0027] Further, for each modification operation event, when it is confirmed that the object of the current modification operation is metadata, if a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot, and the method further includes:

[0028] After the modification operation is completed, a corresponding new uncommitted metadata snapshot is generated;

[0029] Save all uncommitted metadata snapshots in order.

[0030] Furthermore, the method further comprises:

[0031] For each uncommitted transaction,

[0032] If a "rollback" instruction is received, a corresponding undo operation is performed according to the "rollback" instruction;

[0033] If a "commit" indication is received, the last uncommitted metadata snapshot and the closest committed metadata snapshot are aggregated to generate a corresponding new committed metadata snapshot.

[0034] Furthermore, the method further comprises:

[0035] Get the initial committed metadata snapshot and each new committed metadata snapshot and the corresponding timestamp;

[0036] Store the initial committed metadata snapshot,

[0037] All new committed metadata snapshots are stored, or for each new committed metadata snapshot, only metadata that has changed based on the new committed metadata snapshot compared to its closest committed metadata snapshot is stored.

[0038] Furthermore, the method further comprises:

[0039] Determine the point of the metadata snapshot that needs to be restored, and obtain the storage point that stores the metadata snapshot closest to the point;

[0040] The metadata snapshot of the point is generated according to the metadata on the storage point where the metadata snapshot is stored closest to the point and the submitted metadata snapshot associated with the storage point.

[0041] According to another aspect of the present invention, the present invention provides a metadata change synchronization method, which is applied to a target end, and the method comprises:

[0042] Receive statements to be synchronized;

[0043] According to the received statement to be synchronized, the metadata of the target end is updated to be the same as the metadata corresponding to the uncommitted metadata snapshot.

[0044] According to another aspect of the present invention, there is provided a data management method, the method comprising: executing any of the aforementioned metadata change synchronization methods when at least one of capturing, storing, and normalizing changes on a database at a source end.

[0045] According to another aspect of the present invention, the present invention provides a metadata change synchronization system, the system comprising:

[0046] An acquisition module is used to acquire the write-ahead log of the source end to be synchronized, and acquire modification operation events for the source end data in the write-ahead log in real time;

[0047] A parsing module, used to parse the object of the modification operation and the corresponding identification information from each modification operation event;

[0048] The execution module is used to perform the following operations for each modification operation event:

[0049] When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated;

[0050] When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule;

[0051] The generating and sending module is used to generate the statements to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statements to be synchronized to the target end.

[0052] According to another aspect of the present invention, the present invention provides a metadata change synchronization system, which is applied to a target end, and the system includes:

[0053] A receiving module, used for receiving statements to be synchronized;

[0054] The update module is used to update the metadata of the target end to be the same as the metadata corresponding to the uncommitted metadata snapshot according to the received statement to be synchronized.

[0055] According to another aspect of the present invention, the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned metadata change synchronization methods is implemented.

[0056] According to another aspect of the present invention, the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, any of the aforementioned data synchronization methods is implemented.

[0057] Through one or more of the above embodiments of the present invention, at least the following technical effects can be achieved:

[0058] Through the technical solution of the present invention, it is intended to obtain the modification operation events for the source data in the pre-write log before modifying the source data, and parse the corresponding modification operation object and identification information from the modification operation event. In the modification process, different preset rules are used to make corresponding modifications according to the different modification operation objects, and a corresponding uncommitted metadata snapshot is generated. Before the uncommitted metadata snapshot in the above-mentioned uncommitted transaction is committed, any modification to the metadata or data of the source data will not be saved to the downstream storage system. Therefore, when performing metadata or data modification operations in the same transaction, the stored metadata or data will not be modified, but corresponding modifications will be made based on the uncommitted metadata snapshot, thereby ensuring that in the same transaction, if there are metadata changes and data changes, the consistency of the transaction will not be destroyed, and rollback operations can also be supported. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The technical solutions and other beneficial effects of the present invention will be made apparent by describing in detail the specific embodiments of the present invention in conjunction with the accompanying drawings.

[0060] Figure 1 A flowchart of the steps of the metadata change synchronization method provided by an embodiment of the present invention.

[0061] Figure 2 A schematic diagram of a storage structure of metadata changes and data changes capable of time travel provided by an embodiment of the present invention.

[0062] Figure 3 A schematic diagram of the structure of metadata state access and recovery provided by an embodiment of the present invention.

[0063] Figure 4A structural block diagram of a metadata change synchronization system provided by an embodiment of the present invention.

[0064] Figure 5 This is a structural block diagram of another metadata change synchronization system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The technical scheme in the embodiment of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " herein, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0067] A transaction is a set of indivisible operations (logical unit of work). If there is no transaction, various data problems will occur when the database reads, writes, deletes or modifies data, resulting in inconsistent data, invalid data, erroneous data, etc. The four major characteristics of the ACID principle of transactions are briefly described as follows:

[0068] First of all, the atomicity of a transaction means that all operations in a transaction are inseparable as a whole, like an atom. Either all operations succeed or all operations fail. That is, a transaction is either fully executed or not executed at all.

[0069] Secondly, transaction isolation means that multiple concurrently executed transactions are isolated from each other during execution. Transactions cannot interfere with each other and cannot view each other's uncommitted data. The impact on the database is the same as when they are executed serially. For example, if multiple users transfer money to an account at the same time, the final result of the account should be the same as the result of their transfers in sequence.

[0070] Furthermore, transaction durability means that once a transaction is committed, the changes it makes to the data in the database should be permanent, so that all changes in the transaction will not change due to accidents such as power failure, system crash, and race conditions.

[0071] After the above three characteristics are guaranteed, the core consistency of the transaction is achieved, that is, no matter how many transactions are executed, no matter what operations the transactions have, the state of the database is consistent and stable at any time before, during, and after the execution.

[0072] In view of the problem that in the prior art, metadata changes and data changes in a transaction are submitted separately, resulting in failure to support transaction interleaving and failure to support rollback operations, an example is given for illustrative purposes by taking transactions of multiple concurrent operations in a database Wal (Write-ahead log):

[0073] SQL begin; -- transaction 1

[0074] insert into t1(id,name)values(1,'a'); --Transaction 1

[0075] alter t1 add column age int; -- Transaction 1, modify the metadata of table 1, add the age field insert into t1(id,name,age)values(2,'b',10); -- Transaction 1;

[0076] begin; --Transaction 2

[0077] insert into t2(id,name)values(1,'a'); --Transaction 2

[0078] alter t2 add column age int; -- transaction 2, modify the metadata of table 2, add the age field commit; -- transaction 2

[0079] begin; --Transaction 3

[0080] insert into t1(id,name)values(10,'b'); -- Transaction 3, you should see the t1 table structure commit before transaction 1; -- Transaction 3

[0081] insert into t1(id,name,age)values(3,'c',11); --Transaction 1

[0082] rollback; --transaction 1

[0083] begin; --Transaction 4

[0084] insert into t1(id,name)values(2,'b'); -- Transaction 4, you should see the t1 table structure commit before transaction 1; -- Transaction 4

[0085] As shown above, this is the order in which multiple concurrent transactions are finally recorded in the wal log in the database. If the existing metadata synchronization technology does not support transactions, after the change of table t1 is executed, the change of table t1 will be visible to other transactions, and the subsequent rollback operation of transaction 1 will not be executed, which will lead to the following problems:

[0086] 1. Transaction 3 uses the table structure after the change of table t1, and the previous one should be used.

[0087] 2. Because transaction 1 was rolled back, transaction 4 also used the table structure after the change of table t1, and the previous one should be used.

[0088] 3. There is more uncommitted dirty data in table t1.

[0089] Therefore, in order to support the modification of metadata and data simultaneously in the same transaction, it is urgent to provide an efficient and reliable metadata change synchronization method.

[0090] Figure 1 A flowchart of the steps of the metadata change synchronization method provided by an embodiment of the present invention.

[0091] like Figure 1 As shown, the execution subject may be a synchronization tool at the source end, and the metadata change synchronization method includes:

[0092] Step 101: Obtain the write-ahead log of the source end to be synchronized, and obtain modification operation events for source end data in the write-ahead log in real time;

[0093] Step 102: parsing the modification operation object and corresponding identification information from each modification operation event;

[0094] Step 103: For each modification operation event, perform the following operations:

[0095] When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated;

[0096] When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule;

[0097] Step 104: Generate a statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statement to be synchronized to the target end.

[0098] Steps S101 to S104 will be described in detail below.

[0099] In step S101, the write-ahead log of the source end to be synchronized is obtained, and modification operation events for source end data in the write-ahead log are obtained in real time.

[0100] An efficient log algorithm in the database. For non-memory databases, disk I / O operations are a major bottleneck of database efficiency. Under the same amount of data, when a database system using WAL commits a transaction, the disk write operation is only about half of the traditional rollback log, which greatly improves the efficiency of database disk I / O operations and thus improves database performance.

[0101] Exemplarily, in some embodiments, the write-ahead log may include WAL, binlog, etc. Among them, WAL is an important technical means for ensuring the atomicity and persistence of data operations in databases / storage systems, and can also improve the performance of databases in certain scenarios. In databases / storage systems using WAL, all modifications must be written to the log file before submission. Specifically, when writing data or modifying data at the source end, the writing of data or the modification of data will be written to the persistent file of WAL first, thereby ensuring the persistence of our data. If the transaction fails, the record in WAL will be ignored and the modification will be undone; if the transaction succeeds, it will be written back to the database file at a later time and the modification will be submitted. In an embodiment of the present application, the transaction stream to be executed is obtained according to the acquired write-ahead log file of the source end to be synchronized, for example, reading from the beginning of the write-ahead log file from top to bottom, for example, reading down to 1000 lines, 2000 lines, and 3000 lines, and each time a transaction is read in real time, the modification operation event corresponding to the transaction is obtained accordingly to modify the source end data. The source data includes source metadata and source data, so the modification operation event for the source data includes a modification operation event for the metadata and / or a modification operation event for the data.

[0102] In step S102, the object of the modification operation and the corresponding identification information are parsed from each modification operation event;

[0103] In an embodiment of the present application, when parsing the file of the write-ahead log, the parsing module will parse the object of the modification operation and the corresponding identification information from each of the modification operation events. Among them, the object of the modification operation includes, for example, metadata or data; the corresponding identification information of the modification operation includes, for example, one or a combination of the event ID and transaction ID corresponding to each modification operation event. Different transactions will identify different transaction IDs, and the same transaction will use the same transaction ID. And the same transaction may contain modification operation events of the same time with one or more event IDs, or the same transaction may contain modification operation events of different times with multiple event IDs at the same time, or when one transaction is not committed, another transaction also starts to insert a modification operation on the metadata or data of the source end.

[0104] In step 103, for each modification operation event, perform the following operations:

[0105] When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated;

[0106] When it is confirmed that the object of the current modification operation is data, the corresponding modification operation is performed based on the second preset rule.

[0107] Exemplarily, for each modification operation event, the synchronization tool will perform different operations according to the object of the parsed modification operation. For example, when it is confirmed that the object of the current modification operation is metadata, the corresponding modification operation is performed based on the first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated. When it is confirmed that the object of the current modification operation is data, the corresponding modification operation is performed based on the second preset rule.

[0108] In step 104, a statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs is generated, and the statement to be synchronized is sent to the target end.

[0109] The technical solution provided by the embodiment of the present invention is adopted: it is intended to obtain the modification operation events for the source data in the pre-write log before modifying the source data, and parse the corresponding modification operation object and identification information from the modification operation event. In the modification process, different preset rules are used to make corresponding modifications according to the different modification operation objects, and a corresponding uncommitted metadata snapshot is generated. Before the uncommitted metadata snapshot in the above-mentioned uncommitted transaction is committed, any modification to the metadata or data of the source data will not be saved to the downstream storage system. Therefore, when performing metadata or data modification operations in the same transaction, the stored metadata or data will not be modified, but corresponding modifications will be made based on the uncommitted metadata snapshot, thereby ensuring that in the same transaction, if there are metadata changes and data changes, the consistency of the transaction will not be destroyed, and rollback operations can also be supported.

[0110] Even when executing modification operation events with multiple event IDs at the same time or at different times in the same transaction, the metadata saved in the downstream storage system will not be modified. Therefore, when performing multiple metadata or data modification operations in the same transaction, the stored metadata or data will not be modified, but corresponding modifications will be made based on the uncommitted metadata snapshot.

[0111] For example, a current modification operation event of a transaction ID is read, and the modification operation object is parsed to be metadata, and after matching the metadata changes with uncommitted transactions according to the identification information of the current modification operation event, if the corresponding uncommitted transaction cannot be matched, then for the current modification operation event, the system will create a new uncommitted transaction, perform the corresponding modification operation based on the closest committed metadata snapshot, and generate a corresponding uncommitted metadata snapshot after the modification operation is completed. If the corresponding uncommitted transaction can be matched, then for the current modification operation event, the system does not need to re-create a new uncommitted transaction, but performs the corresponding modification operation in the corresponding existing uncommitted transaction based on the closest uncommitted metadata snapshot, and generates a corresponding uncommitted metadata snapshot after the modification operation is completed.

[0112] For example, a current modification operation event of a transaction ID is read, and the modification operation object is parsed to be data, and the data change is matched with an uncommitted transaction according to the identification information of the current modification operation event. If the corresponding uncommitted transaction cannot be matched, then for the current modification operation event, the system will perform the corresponding modification operation based on the closest committed metadata snapshot; if the corresponding uncommitted transaction can be matched, then for the current modification operation event, the system will perform the corresponding modification operation based on the closest uncommitted metadata snapshot.

[0113] It should be noted that in the embodiment of the present invention, out of consideration for the performance of the database / storage system, only when the metadata is parsed to be changed, and after matching the uncommitted transaction according to the identification information of the current modification operation event, if the corresponding uncommitted transaction cannot be matched, a new uncommitted transaction will be created, and a corresponding uncommitted metadata snapshot will be generated in the new uncommitted transaction. When the data is parsed to be changed, even if the corresponding uncommitted transaction cannot be matched according to the identification information of the current modification operation event, a new uncommitted transaction will not be created, but the corresponding data modification operation will be performed based on the closest committed metadata snapshot, thereby reducing the number of uncommitted transactions created, reducing the occupancy of storage and operation resources, and improving the performance of the database / storage system.

[0114] Furthermore, the generation of statements to be synchronized for the uncommitted transaction to which the modification operation belongs includes: generating a metadata change operation statement corresponding to the target end when it is confirmed that the object of the current modification operation is metadata; and generating a data change operation statement corresponding to the target end when it is confirmed that the object of the current modification operation is data.

[0115] For example, in a Postgresql database, when it is confirmed that the object of the current modification operation is metadata, a DDL (Data Definition Language) statement for metadata changes on the corresponding target side is generated. Among them, DDL statements are languages ​​used to describe real-world entities stored in a database. DDL statements are mainly used to define the relationships between real entities, interaction methods, modes, and tables stored in the database, as well as to define data types, links and constraints between tables, and other initialization work. According to the corresponding DDL statements, the target side changes the structure of the target table on the target side to be the same as that of the source table, that is, to have the same metadata information.

[0116] For example, in a Postgresql database, when it is confirmed that the object of the current modification operation is data, a DML (Data Manipulation Language) statement corresponding to the metadata change of the target end is generated. Among them, the DML statement is a data manipulation language, which is used to update, insert, delete and query table data records in the database.

[0117] Furthermore, in order to ensure the consistency of the transaction, the metadata change operation statements and data change operation statements belonging to the same transaction are synchronized to the target end, so that when the target end performs the update operation, the metadata change operation statements and data change operation statements belonging to the same transaction are submitted using the transaction method.

[0118] Figure 2 A schematic diagram of a storage structure of metadata changes and data changes that can time travel provided by an embodiment of the present invention. Figure 2 As shown, the technical solution of the present invention is exemplarily described.

[0119] For example, Figure 2 As shown, when it is parsed from the modification operation event that the metadata of the tA0 table on the source side has changed, the metadata change is matched with the uncommitted transaction according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction cannot be matched, a new uncommitted transaction is created at the corresponding time 1, for example, uncommitted transaction 1, and the corresponding modification operation is performed in the uncommitted transaction 1 based on the closest committed metadata snapshot, and after the tA0 table is changed to the tA1 table, a corresponding uncommitted metadata snapshot 1 is generated, which can also be referred to as the uncommitted metadata snapshot 1 corresponding to the uncommitted transaction 1.

[0120] It should be understood that for each uncommitted transaction, when generating a corresponding uncommitted metadata snapshot, it is necessary to obtain the status information of the current transaction, including the transaction ID, transaction start time, participant list, participant status, etc. This information is then integrated into a snapshot for monitoring and management during the transaction execution process.

[0121] When it is parsed from the modification operation event that metadata changes have occurred in the tB0 table on the source side, the metadata changes are matched with the uncommitted transactions according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction cannot be matched, a new uncommitted transaction is created at the corresponding time 2, for example, uncommitted transaction 2, and the corresponding modification operation is performed in the uncommitted transaction 2 based on the closest committed metadata snapshot, and after the tB0 table is changed to the tB1 table, a corresponding uncommitted metadata snapshot 1 is generated, which can also be referred to as the uncommitted metadata snapshot 1 corresponding to the uncommitted transaction 2.

[0122] Furthermore, for each modification operation event, when it is confirmed that the object of the current modification operation is metadata, if a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the uncommitted metadata snapshot that is closest to it. The method also includes: after the modification operation is completed, generating a corresponding new uncommitted metadata snapshot; and saving all uncommitted metadata snapshots in sequence.

[0123] For example, continue to refer to Figure 2 As shown, when it is parsed from the modification operation event that the metadata change occurs in the tC0 table on the source side, the metadata change is matched with the uncommitted transaction according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction 1 can be matched, the corresponding modification operation is continued to be performed in the corresponding uncommitted transaction 1 based on the uncommitted metadata snapshot that is closest to it, and after the tC0 table is changed to the tC1 table, a corresponding uncommitted metadata snapshot 2 is generated at the corresponding time 3. Here, it can also be referred to as the uncommitted metadata snapshot 2 corresponding to the uncommitted transaction 1, that is, the uncommitted metadata snapshots corresponding to the uncommitted transaction 1 are stored in sequence in the cache.

[0124] It should be noted that, in the embodiment of the present invention, in the same uncommitted transaction, although multiple uncommitted metadata snapshots may be generated based on multiple changes in metadata, if the generated uncommitted metadata snapshots are not committed, the metadata changes will not be written to the target database.

[0125] Furthermore, the method also includes: for each uncommitted transaction, if a "rollback" instruction is received, a corresponding undo operation is performed according to the "rollback" instruction; if a "commit" instruction is received, the current last uncommitted metadata snapshot and the closest committed metadata snapshot are aggregated to generate a corresponding new committed metadata snapshot.

[0126] For example, in some embodiments, for each uncommitted transaction, if a "Roll Back" indication is received, a corresponding undo operation is performed according to the "Roll Back" indication; if a "Commit" indication is received, the current last uncommitted metadata snapshot and the closest committed metadata snapshot are aggregated to generate a corresponding new committed metadata snapshot.

[0127] For example, continue to refer to Figure 2 As shown, for the uncommitted transaction 2, in response to receiving the "commit" instruction, the current last uncommitted metadata snapshot 1 in the uncommitted transaction 2 and the closest committed metadata snapshot are aggregated to generate a corresponding new committed metadata snapshot 2 at time 4. For the uncommitted transaction 1, in response to receiving the "commit" instruction, the current last uncommitted metadata snapshot 2 in the uncommitted transaction 1 and the closest committed metadata snapshot 2 are aggregated to generate a corresponding new committed metadata snapshot 3 at time 5. At this time, the tA0 table, tB0 table, and tC0 table have all been changed to the new tA1 table, tB1 table, and tC1 table. In addition, the data of the corresponding tA1 table, tB1 table, and tC1 table are also changed to be the same as the data of the table indexed by the committed metadata snapshot 3.

[0128] In some embodiments, Figure 2 As shown, exemplarily, when it is parsed from the modification operation event that the corresponding data changes occurred in the tA0 table on the source side before time 1, the data changes are matched with the uncommitted transactions according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction cannot be matched, the corresponding modification operation can be performed only based on the closest committed metadata snapshot.

[0129] Exemplarily, when it is parsed from the modification operation event that the corresponding data changes occurred in the tB0 table on the source side before time 2, the data changes are matched with the uncommitted transactions according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction cannot be matched, the corresponding modification operation can be performed only based on the closest committed metadata snapshot.

[0130] Exemplarily, when it is parsed from the modification operation event that the corresponding data changes occurred in the tB0 table on the source side before time 4, the data changes are matched with the uncommitted transactions according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction 2 can be matched, and the uncommitted transaction 2 is still in an uncommitted state, at this time, the corresponding modification operation is performed only based on the closest committed metadata snapshot in the uncommitted transaction 2.

[0131] Exemplarily, when it is parsed from the modification operation event that corresponding data changes occurred in the tA1 table and the tC1 table in the modification operation event before time 5, since the current transaction is still in an uncommitted state, the data change is matched with the uncommitted transaction according to the identification information of the current modification operation event. Since the corresponding uncommitted transaction 1 can be matched, at this time, the corresponding modification operation is performed only based on the uncommitted metadata snapshot that is closest to it in the uncommitted transaction 1.

[0132] For example, when it is analyzed that the corresponding data changes occurred in the tA1 table, tB1 table and tC1 table after time 5 in the modification operation event, since the current transaction is in the committed state, the corresponding modification operation can be directly executed based on the closest committed metadata snapshot 3.

[0133] In other embodiments, when a modification operation event is parsed to find that there is a modification operation event record between the uncommitted metadata snapshot 1 and the uncommitted metadata snapshot 2 in the uncommitted transaction 1, this record is not a metadata change, for example, for the table tA1, it inserts a piece of data, but the inserted piece of data is inserted according to the new table structure of the tA1 table after the change. This is equivalent to using the latest uncommitted metadata snapshot of the table before the data change to modify the data of the table.

[0134] Figure 3 A schematic diagram of the structure of metadata state access and recovery provided by an embodiment of the present invention.

[0135] For example, Figure 3 As shown, regarding the state storage of metadata, the method includes:

[0136] Get the initial committed metadata snapshot and each new committed metadata snapshot and the corresponding timestamps;

[0137] Store the initial committed metadata snapshot,

[0138] All new submitted metadata snapshots are stored, that is, full storage is performed; or in order to save storage space, for each new submitted metadata snapshot, only the metadata that has changed based on the new submitted metadata snapshot compared to its most recent submitted metadata snapshot is stored.

[0139] For example, Figure 3 As shown, regarding the recovery of the metadata state, the method further includes:

[0140] Determine the point of the metadata snapshot that needs to be restored, and obtain the storage point that stores the metadata snapshot closest to the point;

[0141] The metadata snapshot of the point is generated according to the metadata on the storage point where the metadata snapshot is stored closest to the point and the submitted metadata snapshot associated with the storage point.

[0142] Specifically, in the embodiment of the present application, the initial submitted metadata snapshot and the new submitted metadata snapshot are stored on the hard disk. When storing the new submitted metadata snapshot, in order to save storage space, it is not necessary to store the entire snapshot of the submitted metadata snapshot generated at each time node in the backend storage. Only the objects with changes in the snapshot need to be stored in the backend. All subsequent metadata changes are based on the snapshot changes.

[0143] For example, Figure 3 As shown, when the point to be replayed is between time 2 and time 3 stored in the disk, first obtain the storage point that stores the metadata snapshot closest to the playback point, that is, obtain the tB1 table structure corresponding to the storage point at time 2, and then, according to the tA1 table structure and tC0 table structure missing at the storage point at time 2, respectively index forward, for example, index forward to the tA1 table structure corresponding to the storage point at time 1, and index forward to the tC0 table structure corresponding to the storage point at the initial moment. Finally, splicing processing is performed based on the tB1 table structure of the storage point at time 2, the tA1 table structure of the storage point at time 1, and the tC0 table structure of the storage point at the initial moment to obtain the metadata snapshot of the point that currently needs to be replayed.

[0144] According to another aspect of the present invention, a metadata change synchronization method is also provided, which is applied to a target end, and the method comprises:

[0145] Receive statements to be synchronized;

[0146] According to the received statement to be synchronized, the metadata of the target end is updated to be the same as the metadata corresponding to the uncommitted metadata snapshot.

[0147] The target end changes the structure of the target table on the target end to be the same as the structure of the source table according to the corresponding statement to be synchronized, that is, to have the same metadata information.

[0148] Exemplarily, the target end may update the metadata of the target end to be the same as the metadata corresponding to the uncommitted metadata snapshot according to the received DDL statements and DML statements belonging to the same transaction, thereby achieving transaction-level changes in the metadata.

[0149] It should be understood that other aspects and effects of the method can be found in the aforementioned metadata change synchronization method, and will not be described in detail here.

[0150] According to another aspect of the present invention, a data synchronization method is provided, the method comprising:

[0151] When at least one of capturing, storing, and normalizing the changes in the database at the source end is performed, the metadata change synchronization method described in any of the above-mentioned implementations is executed.

[0152] Figure 4 A structural block diagram of a metadata change synchronization system provided by an embodiment of the present invention.

[0153] like Figure 4 According to another aspect of the present invention, a metadata change synchronization system is provided, the system 200 comprising:

[0154] The acquisition module 210 is used to acquire the write-ahead log of the source end to be synchronized, and acquire the modification operation events for the source end data in the write-ahead log in real time;

[0155] The parsing module 220 is used to parse the object of the modification operation and the corresponding identification information from each modification operation event;

[0156] The execution module 230 is used to perform the following operations for each modification operation event:

[0157] When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated;

[0158] When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule;

[0159] The generating and sending module 240 is used to generate a statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statement to be synchronized to the target end.

[0160] Exemplarily, the metadata change operation statement and the data change operation statement belonging to the same transaction are synchronized to the target end.

[0161] It should be understood that other aspects and effects of the system can be found in the embodiment of the aforementioned metadata change synchronization method, and will not be described in detail here.

[0162] Figure 5 This is a structural block diagram of another metadata change synchronization system provided by an embodiment of the present invention.

[0163] like Figure 5 According to another aspect of the present invention, a metadata change synchronization system is provided, the system 300 comprising:

[0164] A receiving module 310, for receiving statements to be synchronized;

[0165] The updating module 320 is used to update the metadata of the target end to be the same as the metadata corresponding to the uncommitted metadata snapshot according to the received statement to be synchronized.

[0166] The target end changes the structure of the target table on the target end to be the same as the structure of the source table according to the corresponding statement to be synchronized, that is, to have the same metadata information.

[0167] It should be understood that other aspects and effects of the system can be found in the embodiment of the aforementioned metadata change synchronization method, and will not be described in detail here.

[0168] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for synchronizing metadata changes in any of the embodiments described above is implemented.

[0169] The specific definition and implementation of the above steps can be found in the embodiment of the metadata change synchronization method, which will not be described in detail here.

[0170] In another embodiment, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium, and when the computer program is executed by a processor, any of the aforementioned data synchronization methods is implemented.

[0171] The specific definition and implementation of the above steps can be found in the embodiments of the data management method, which will not be described in detail here.

[0172] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced

[0173] SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM), etc.

[0174] In summary, although the present invention has been disclosed as above in terms of preferred embodiments, the above preferred embodiments are not intended to limit the present invention. A person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope defined in the claims.

Claims

1. A metadata change synchronization method, characterized in that: The method comprises: Obtain the write-ahead log of the source end to be synchronized, and obtain the modification operation events for the source end data in the write-ahead log in real time; Parsing the object of the modification operation and the corresponding identification information from each modification operation event; For each modification operation event, perform the following operations: When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated; When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule; Generate a statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statement to be synchronized to the target end.

2. The method according to claim 1, characterized in that For each modification operation event, when it is confirmed that the object of the current modification operation is metadata, executing the corresponding modification operation based on the first preset rule includes: Matching the metadata changes with uncommitted transactions according to the identification information, If no corresponding uncommitted transaction is matched, a new uncommitted transaction is created, and the corresponding modification operation is performed based on the closest committed metadata snapshot; If a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot.

3. The method according to claim 1, characterized in that For each of the modification operation events, when it is confirmed that the object of the current modification operation is data, executing the corresponding modification operation based on the second preset rule includes: Matching the data changes with uncommitted transactions according to the identification information, If no corresponding uncommitted transaction is matched, the corresponding modification operation is performed based on the closest committed metadata snapshot; If a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot.

4. The method according to claim 1, characterized in that The generating of the statement to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs includes: When it is confirmed that the object of the current modification operation is metadata, a metadata modification operation statement corresponding to the target end is generated; When it is confirmed that the object of the current modification operation is data, a data change operation statement corresponding to the target end is generated.

5. The method according to claim 4, characterized in that The method further comprises: Synchronize metadata change operation statements and data change operation statements belonging to the same transaction to the target end.

6. The method according to claim 1, characterized in that The identification information includes one or a combination of an event ID and a transaction ID.

7. The method according to claim 2, characterized in that For each modification operation event, when it is confirmed that the object of the current modification operation is metadata, if a corresponding uncommitted transaction can be matched, the corresponding modification operation is performed based on the closest uncommitted metadata snapshot, and the method further includes: After the modification operation is completed, a corresponding new uncommitted metadata snapshot is generated; Save all uncommitted metadata snapshots in order.

8. The method according to claim 1, characterized in that The method further comprises: For each uncommitted transaction, If a "rollback" instruction is received, a corresponding undo operation is performed according to the "rollback" instruction; If a "commit" instruction is received, the last uncommitted metadata snapshot and the closest committed metadata snapshot are aggregated to generate a corresponding new committed metadata snapshot.

9. The method according to claim 8, characterized in that The method further comprises: Get the initial committed metadata snapshot and each new committed metadata snapshot and the corresponding timestamp; Store the initial committed metadata snapshot, All new committed metadata snapshots are stored, or for each new committed metadata snapshot, only metadata that has changed based on the new committed metadata snapshot compared to its closest committed metadata snapshot is stored.

10. The method according to claim 9, characterized in that The method further comprises: Determine the point of the metadata snapshot that needs to be restored, and obtain the storage point that stores the metadata snapshot closest to the point; The metadata snapshot of the point is generated according to the metadata on the storage point where the metadata snapshot is stored closest to the point and the submitted metadata snapshot associated with the storage point.

11. A metadata change synchronization method, applied to a target end, characterized in that: The method comprises: Receive statements to be synchronized; According to the received statement to be synchronized, the metadata of the target end is updated to be the same as the metadata corresponding to the uncommitted metadata snapshot.

12. A data synchronization method, characterized in that: The method comprises: When at least one of capturing, storing, and normalizing changes in a database at a source end is performed, the metadata change synchronization method described in any one of claims 1 to 10 is executed.

13. A metadata change synchronization system, characterized in that: The system comprises: An acquisition module is used to acquire the write-ahead log of the source end to be synchronized, and acquire modification operation events for the source end data in the write-ahead log in real time; A parsing module, used to parse the object of the modification operation and the corresponding identification information from each modification operation event; The execution module is used to perform the following operations for each modification operation event: When it is confirmed that the object of the current modification operation is metadata, a corresponding modification operation is performed based on a first preset rule, and after the modification operation is completed, a corresponding uncommitted metadata snapshot is generated; When it is confirmed that the object of the current modification operation is data, executing a corresponding modification operation based on a second preset rule; The generating and sending module is used to generate the statements to be synchronized corresponding to the uncommitted transaction to which the modification operation belongs, and send the statements to be synchronized to the target end.

14. A metadata change synchronization system, applied to a target end, characterized in that: The system comprises: A receiving module, used for receiving statements to be synchronized; The update module is used to update the metadata of the target end to be the same as the metadata corresponding to the uncommitted metadata snapshot according to the received statement to be synchronized.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the metadata change synchronization method according to any one of claims 1 to 10 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the metadata change synchronization method according to claim 11 is implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data synchronization method according to claim 12 is implemented.

Images