A Distributed Database Metadata Replication Method and System
By employing a distributed database metadata replication method, and utilizing the KVSender, KVRewriter, and Executor modules, real-time synchronization and rewriting of metadata are achieved. This solves the problems of real-time performance and DDL operation support between database clusters, enabling flexible metadata synchronization and data replication.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 上海沄熹科技有限公司
- Filing Date
- 2023-07-27
- Publication Date
- 2026-05-26
Smart Images
Figure CN116991848B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed database technology, specifically a distributed database metadata replication method and system. Background Technology
[0002] Data replication between databases primarily takes the form of file transfers or master-slave replication. Each method has its advantages and disadvantages. File transfers are suitable for importing and exporting the entire database from a backup. This method is simple to operate and can import large amounts of data in a short time. However, this method cannot meet real-time requirements; it requires pausing the database or exporting a specific checkpoint for import / export.
[0003] While a primary-standby approach can meet most real-time requirements, it is not well-suited to handling changes in system metadata and cannot perform fine-tuning of the primary and standby databases. For example, copying data table t1 from the primary database to data table t2 on the standby database often requires implementation by the upper-level database application, which is inflexible. Additionally, it has poor support for DDL operations between the primary and standby databases. Summary of the Invention
[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a distributed database metadata replication method and system. This system can replicate metadata between different database clusters while simultaneously replicating business data in real time. This ensures real-time performance and supports metadata synchronization between the primary and backup databases, without affecting the replication of normal data.
[0005] The technical solution adopted by this invention to solve its technical problem is:
[0006] A distributed database metadata replication method is proposed, the implementation of which includes a source-side sending module KVSender, a target-side rewriting module KVRewriter, and a target-side execution and disk persistence module Executor.
[0007] The source-side sending module KVSender is used to send the data that needs to be synchronized from the source database to the target database.
[0008] The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value.
[0009] The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk.
[0010] This method provides a dedicated connection for synchronizing metadata, enabling DDL statements that operate on tables to be copied in real time between different databases. On the receiving end, a dedicated KVRewriter is provided to rewrite metadata for identical tables between the two clusters. No other connections are needed between the two clusters besides the data transfer connection, and no file import / export transfers are required. The KVRewriter can be dynamically configured according to the different types of metadata.
[0011] The proposed metadata replication method can replicate business data in real time while also replicating metadata between different database clusters. This ensures real-time performance and supports metadata synchronization between the primary and backup databases, without affecting the replication of normal data.
[0012] Preferably, in the database metadata, if a user creates table t1, then system.descriptor stores the table structure of all data tables, system.namespace stores the mapping relationship between the user table namespace and the data table, system.comment stores the comment information of the user table, and the user table t1 stores the corresponding user data.
[0013] Preferably, the source-end sending module KVSender includes a connection module Connector, a kv filtering module Filter, and a kv forwarding module Sender. The user sets the IP address and certificate directory of the target end so that the source end can establish a TCP connection to the target end.
[0014] The key-value (KV) filtering module (Filter) iterates through all KV operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and replicated. For a KV database, all user DDL operations are still KV operations, and these DDL statements generate metadata KV operations. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module (Filter) passes the filtered key that needs to be sent to the KV forwarding module (Sender), and the KV forwarding module (Sender) sends the metadata KV to the backup end through a pipe.
[0015] Preferably, the target-side rewrite module KVRewriter always stores the latest copy of the corresponding source-side TableDescriptor, as well as all fields that need to be rewritten, and stores the rewrite mapping relationship.
[0016] Preferably, the target end rewrite module KVRewriter receives different key values depending on the metadata to be synchronously copied. For system.comment and system.namespace, keyWriter only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue.
[0017] For TableDescriptor, the target-side rewrite module KVRWriter needs to rewrite TableID, as well as the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution.
[0018] Preferably, the target-end execution and disk writing module Executor performs real-time replication between the source and target ends. When key-value pairs are being transferred, the transaction in which the operation is taking place may not have been committed yet. The Executor will wait for the transaction to be committed before writing all key-value pairs of the transaction to storage, or it will discard all key-value pairs of the transaction after receiving transaction rollback information.
[0019] By providing a dedicated metadata replication channel for the distributed database system, rather than through master-slave or file import / export replication methods, real-time performance is ensured while allowing selection and switching between different distributed database clusters. Because a separate data replication channel is provided between different clusters, each database cluster is independent of the others. Users can selectively replicate metadata information; with appropriate settings, they can selectively synchronize and replicate metadata for specific data tables or system tables. DDL operations can also be replicated simultaneously with table data replication.
[0020] This invention also claims a distributed database metadata replication system, comprising a source-side sending module KVSender, a target-side rewriting module KVRewriter, and a target-side execution and disk persistence module Executor.
[0021] The source-side sending module KVSender sends the data that needs to be synchronized from the source database to the target database.
[0022] The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value.
[0023] The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk.
[0024] Furthermore, the source-end sending module KVSender includes a connection module Connector, a kv filtering module Filter, and a kv forwarding module Sender. The user sets the target IP address and certificate directory so that the source end can establish a TCP connection to the target end.
[0025] The key-value (KV) filtering module (Filter) iterates through all KV operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and replicated. For the KV database, all user DDL operations are still KV operations, and these DDL statements generate metadata KV operations. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module (Filter) passes the filtered key that needs to be sent to the KV forwarding module (Sender), and the KV forwarding module (Sender) sends the metadata KV to the backup end through a pipe.
[0026] Furthermore, in the database metadata, if a user creates table t1, then system.descriptor stores the table structure of all data tables, system.namespace stores the mapping relationship between the user table namespace and the data table, system.comment stores the comment information of the user table, and the user table t1 stores the corresponding user data.
[0027] The target end rewrite module KVRewriter receives different key values depending on the metadata to be synchronously copied. For system.comment and system.namespace, keyWriter only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue.
[0028] For TableDescriptor, in addition to rewriting TableID, keyWriter also needs to rewrite the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution.
[0029] Furthermore, in the target-end execution and disk write module Executor, since the replication between the source and target ends is performed in real time, when the key-value pairs are being transferred, the transaction in which the operation is taking place may not have been committed yet. The Executor will wait until the transaction is committed before writing all the key-value pairs of the transaction to storage, or until it receives transaction rollback information and discards all the key-value pairs of the transaction.
[0030] Compared with the prior art, the distributed database metadata replication method and system of the present invention have the following advantages:
[0031] This method and system can synchronize data tables between different database clusters;
[0032] This method and system innovatively propose a method specifically for metadata replication, which can simultaneously replicate DDL operations on the table, such as creating indexes, modifying primary keys, modifying columns, and renaming, while replicating table data in real time.
[0033] The proposed method and system for metadata synchronization between different database clusters can adjust the synchronization effect on the target end according to different mapping relationships. Attached Figure Description
[0034] Figure 1 This is a structural diagram of a metadata copying method provided in an embodiment of the present invention;
[0035] Figure 2 This is a structural diagram of the source-end sending module KVSender provided in one embodiment of the present invention;
[0036] Figure 3 This is a structural diagram of the target-side rewriting module KVRewriter provided in one embodiment of the present invention;
[0037] Figure 4 This is a structural diagram of the target end disk writing module Executor provided in one embodiment of the present invention. Detailed Implementation
[0038] The present invention will be further described below with reference to specific embodiments.
[0039] This invention provides a distributed database metadata replication method, which includes a source-side sending module KVSender, a target-side rewriting module KVRewriter, and a target-side execution and disk persistence module Executor.
[0040] The source-side sending module KVSender is used to send the data that needs to be synchronized from the source database to the target database.
[0041] The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value.
[0042] The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk.
[0043] In a typical distributed key-value database, all user data, as well as the database's metadata, are encoded using key-value pairs. Here, user data refers to the data tables created by the user, while metadata refers to the system tables used for user operations. In the database metadata, assuming a user creates table t1, `system.descriptor` stores the table structure of all data tables, `system.namespace` stores the mapping between user table namespaces and data tables, and `system.comment` stores the comment information for user tables. User table t1 stores the corresponding user data. If, during data table replication, only the data tables are copied without copying the metadata, subsequent DDL operations such as index creation, data backfilling, table renaming, and adding / deleting columns will not be synchronized, or may result in data inconsistency.
[0044] This method is as follows: Figure 1 As shown, metadata replication is achieved through the source-side sending module KVSender, the target-side rewriting module KVRewriter, and the target-side execution and disk persistence module Executor. The specific implementation is as follows:
[0045] The function of the source-side sending module KVSender is to send the data that needs to be synchronized from the source database to the target database. For example... Figure 2 As shown, it includes a Connector module, a Filter module, and a Sender module. Users configure the target's IP address and certificate directory to enable the source to establish a TCP connection to the target.
[0046] The key-value (KV) filtering module (Filter) iterates through all KV operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and replicated. For a KV database, all user DDL operations are still KV operations, and these DDL statements generate metadata KV operations. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module (Filter) passes the filtered key that needs to be sent to the KV forwarding module (Sender), and the KV forwarding module (Sender) sends the metadata KV to the backup end through a pipe.
[0047] The function of the target-side rewrite module KVRewriter is to rewrite the sent key and value into the correct key and value.
[0048] Although the table structure and data of the user table are consistent on both the source and backup sides, the table descriptor of the user table differs between the source and target sides. Since TableID requests between the two different clusters are independent, TableID, foreign key constraints, etc., need to be rewritten. The target-side rewrite module KVRewriter always maintains the latest copy of the corresponding source-side TableDescriptor, as well as all fields that need to be rewritten, and saves the rewrite mapping relationships.
[0049] like Figure 3 As shown, the key-value pairs received by the target rewrite module KVRewriter differ depending on the metadata to be synchronized and copied. For system.comment and system.namespace, keyWriter only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue.
[0050] For TableDescriptor, the target-side rewrite module KVRWriter needs to rewrite TableID, as well as the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution.
[0051] The Executor module on the target end is used to resolve transaction dependencies and persist data to disk. Since replication between the source and target ends is performed in real-time, when key-value pairs are transferred, the transaction in which the operation takes place may not yet have been committed. The Executor waits until the transaction is committed before writing all key-value pairs for that transaction to storage, or receives transaction rollback information and discards all key-value pairs from that transaction. Figure 4 As shown.
[0052] This method provides a dedicated metadata replication channel for the distributed database system, rather than relying on master-slave or file import / export replication methods. This ensures real-time performance while allowing selection and switching between different distributed database clusters. Because a separate data replication channel is provided between different clusters, each database cluster is independent. Users can choose to replicate metadata information; with appropriate settings, they can selectively synchronize and replicate metadata for specific data tables or system tables. DDL operations can also be replicated simultaneously with table data replication.
[0053] This invention also provides a distributed database metadata replication system, which includes a source-side sending module KVSender, a target-side rewriting module KVRewriter, and a target-side execution and disk persistence module Executor.
[0054] The source-side sending module KVSender sends the data that needs to be synchronized from the source database to the target database.
[0055] The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value.
[0056] The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk.
[0057] The source-end sending module KVSender includes a connection module Connector, a kv filtering module Filter, and a kv forwarding module Sender. The user sets the target IP address and certificate directory so that the source end can establish a TCP connection to the target end.
[0058] The key-value (KV) filtering module (Filter) iterates through all KV operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and replicated. For the KV database, all user DDL operations are still KV operations, and these DDL statements generate metadata KV operations. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module (Filter) passes the filtered key that needs to be sent to the KV forwarding module (Sender), and the KV forwarding module (Sender) sends the metadata KV to the backup end through a pipe.
[0059] In the database metadata, if a user creates table t1, then system.descriptor stores the table structure of all data tables, system.namespace stores the mapping relationship between the user table namespace and the data table, system.comment stores the comment information of the user table, and the user table t1 stores the corresponding user data.
[0060] The target-side rewrite module, KVRewriter, rewrites the sent key and value into correct key and value. Although the table structure and data of the user table are consistent on both the source and backup sides, the table descriptor of the user table differs between the source and target sides. Since TableID requests between the two different clusters are independent, rewriting of TableID, foreign key constraints, etc., is necessary. KVRewriter always maintains the latest copy of the corresponding source-side TableDescriptor, as well as all fields that need to be rewritten, and saves the rewrite mapping relationship.
[0061] The key-value pairs received by the target rewrite module KVRewriter differ from the metadata to be synchronized and copied. For system.comment and system.namespace, keyWriter only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue.
[0062] For TableDescriptor, in addition to rewriting TableID, keyWriter also needs to rewrite the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution.
[0063] The Executor module on the target side is used to resolve transaction dependencies and write data to disk. Since the replication between the source and target sides is performed in real time, when key-value pairs are transferred between them, the transaction in which the operation is taking place may not have been committed yet. The Executor will wait until the transaction is committed before writing all the key-value pairs of the transaction to storage, or until it receives transaction rollback information and discards all the key-value pairs of the transaction.
[0064] This system provides a dedicated connection for synchronizing metadata, enabling DDL statements that operate on tables to be copied in real time between different databases. On the receiving end, a unique KVRewriter is provided to rewrite metadata for identical tables between the two clusters. No other connections are needed between the two clusters besides the data transfer connection, and no file import / export transfers are required. KVRewriter can be dynamically configured according to the different types of metadata.
[0065] This system can replicate business data in real time and also replicate metadata between different database clusters. This ensures real-time performance while supporting metadata synchronization between the primary and backup databases and the target database, without affecting the replication of normal data.
[0066] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.
[0067] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.
Claims
1. A distributed database metadata replication method, characterized in that, The implementation of this method includes a source-side sending module KVSender, a target-side rewriting module KVRewriter, and a target-side execution and disk persistence module Executor. The source-side sending module KVSender is used to send the data that needs to be synchronized from the source database to the target database. The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value. The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk. The source-end sending module KVSender includes a connection module Connector, a kv filtering module Filter, and a kv forwarding module Sender. The user sets the IP address and certificate directory of the target end so that the source end can establish a TCP connection to the target end. The key-value (KV) filtering module Filter iterates through all key-value operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and copied. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module Filter passes the filtered key that needs to be sent to the KV forwarding module Sender, and the KV forwarding module Sender sends the metadata key-value to the backup end through the pipeline. In the database metadata, if a user creates table t1, then system.descriptor stores the table structure of all data tables, system.namespace stores the mapping relationship between the user table namespace and the data table, system.comment stores the comment information of the user table, and the user table t1 stores the corresponding user data. The target-side rewrite module KVRewriter, for system.comment and system.namespace, only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue. For TableDescriptor, in addition to rewriting TableID, the target-side rewrite module KVRWriter also needs to rewrite the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution. The target-side execution and disk write module Executor performs real-time replication between the source and target ends. When key-value pairs are being transferred between them, the transaction in which the operation is taking place may not have been committed yet. The Executor will wait until the transaction is committed before writing all key-value pairs of the transaction to storage, or until it receives transaction rollback information and discards all key-value pairs of the transaction.
2. The distributed database metadata replication method according to claim 1, characterized in that, The target-side rewrite module KVRewriter always stores the latest copy of the corresponding source-side TableDescriptor, as well as all fields that need to be rewritten, and saves the rewrite mapping relationship.
3. A distributed database metadata replication system, characterized in that, The implementation includes the source-side sending module KVSender, the target-side rewriting module KVRewriter, and the target-side execution and disk persistence module Executor. The source-side sending module KVSender is used to send the data that needs to be synchronized from the source database to the target database. The target-side rewrite module KVRewriter rewrites the sent key and value into the correct key and value. The Executor module on the target side is used to resolve transaction dependencies and perform data persistence on disk. The source-end sending module KVSender includes a connection module Connector, a kv filtering module Filter, and a kv forwarding module Sender. The user sets the target IP address and certificate directory so that the source end can establish a TCP connection to the target end. The KV filtering module Filter iterates through all KV operations that need to be synchronized. When the user sets the corresponding system table to be synchronized, the source end filters the TableID of the data table to be synchronized and copied. The KV filter parses the metadata key, filters out the original TableID, and determines whether it needs to be sent. The KV filtering module Filter passes the filtered key that needs to be sent to the KV forwarding module Sender. The KV forwarding module Sender sends the metadata KV to the backup end through the pipeline. In the database metadata, if a user creates table t1, then system.descriptor stores the table structure of all data tables, system.namespace stores the mapping relationship between the user table namespace and the data table, system.comment stores the comment information of the user table, and the user table t1 stores the corresponding user data. The target-side rewrite module KVRewriter, for system.comment and system.namespace, only needs to decode the key, parse it to obtain the original TableID, replace it with the new TableID according to the rewrite mapping relationship stored in keyWriter, and re-encode it to complete the rewriting of the metadata KeyValue. For TableDescriptor, in addition to rewriting TableID, the target-side rewrite module KVRWriter also needs to rewrite the database, schema, and foreign key dependencies of the user table on the target side according to the mapping relationship; after completion, keyWriter hands over the rewritten key-value pair to Executor for execution. The target-side execution and disk write module Executor performs real-time replication between the source and target ends. When key-value pairs are being transferred between them, the transaction in which the operation is taking place may not have been committed yet. The Executor will wait until the transaction is committed before writing all key-value pairs of the transaction to storage, or until it receives transaction rollback information and discards all key-value pairs of the transaction.