A distributed database query system, method, device and medium supporting multi-copy consistent reading
By introducing multi-replica consistent reading module and global transaction read consistency module in the distributed database query system, non-strict distributed read consistency level and backup copy selection strategy are adopted, and the problem of multi-replica read query delay is solved, and the implementation of the implementation of the Efficient global transaction read consistency and data version freshness, improving system performance.
Patent Information
- Application Number
- CN202410283694.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-03-13
AI Technical Summary
When the prior art ensures the global transaction consistency of multiple replica reads, it leads to a long query delay, especially when the main replica write pressure of a stand-alone database is high, the synchronization speed of the backup replica data cannot keep up with the main replica update speed. Causes increased query delay.
By introducing multi-replica consistency reading module, global transaction read consistency module and multi-replica read management module in the distributed database query system, non-strict distributed read consistency level and backup copy selection strategy are adopted to store from each shard Select the final replica from multiple copies of a stand-alone database to meet the requirements of global transaction read consistency and data version close to the latest, and avoid waiting for data synchronization.
It shortens the overall query delay, improves the performance of distributed databases, ensures global transaction read consistency and data version freshness, and reduces the reading pressure of the main replica of a stand-alone database.
Smart Images

Figure CN118093647B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the technical field of distributed databases, and in particular to a distributed database query system, method, device and medium that supports multi-copy consistent reading. Background Art
[0002] A distributed database is a database that decomposes the business's Structured Query Language (SQL) requests to multiple shards for execution on a single-machine database. A single-machine database can usually guarantee the ACID (atomicity, consistency, isolation, and durability) characteristics, but simply decomposing the business SQL request into multiple requests for execution on a single-machine database cannot guarantee the isolation of the distributed transaction as a whole, that is, it cannot guarantee global transaction read consistency.
[0003] The prior art provides a multi-version concurrency control method based on a global clock service, but this method only solves the global transaction consistency of reading from the primary copy of a stand-alone database in each shard storage, and reading from the backup copy of a stand-alone database in each shard storage cannot guarantee the global transaction read consistency. In order to ensure the global transaction consistency of multi-copy reading, the existing main method is as follows: the client sends a request to the entrance Gate of the distributed database; Gate will recognize that the request will be sent to the shard storage storing data, and will first obtain the maximum commit timestamp max_cts of the current committed transaction from the primary copy of the stand-alone database in the shard storage, and then obtain the snapshot timestamp sts from the global clock service; max_cts and sts are sent together with the request to a backup copy of the stand-alone database in the shard storage; the backup copy of the stand-alone database determines whether to wait until the data is synchronized to the corresponding location based on the received max_cts; then the data visibility is determined based on sts, and the result is returned to the Gate. However, the above method has performance issues. When the write pressure of the primary copy of the stand-alone database is relatively large, the data synchronization speed of the backup copy cannot keep up with the update speed of the primary copy data. It takes a long time to read data from the backup copy until the data is synchronized to the corresponding location, resulting in a long query delay. Summary of the invention
[0004] The present invention provides a distributed database query system, method, device and medium supporting multi-copy consistent reading, so as to solve the problem of long query delay caused by the prior art.
[0005] According to one aspect of the present invention, a distributed database query system supporting multi-copy consistent reading is provided, comprising: a multi-copy consistent reading module, a global transaction read consistency module and a multi-copy read management module;
[0006] The multi-copy consistency read module is used to provide at least one non-strict distributed read consistency level;
[0007] The global transaction read consistency module is used to ensure the global transaction read consistency;
[0008] The multi-copy read management module is used to select a final copy from multiple copies of a stand-alone database stored in each shard to read data after the global transaction read consistency module ensures the global transaction read consistency, based on the shard storage list, the non-strict distributed read consistency level and the backup copy selection strategy. The final backup copy meets the requirements of global transaction read consistency and data version being close to the latest.
[0009] According to another aspect of the present invention, a distributed database query method supporting multi-copy consistent reading is provided, comprising:
[0010] After receiving the structured query statement sent by the user, the distributed database entrance parses the structured query statement and determines the shard storage list where the structured query statement to be queried is distributed;
[0011] If multi-replica read is used, obtain the backup replica list and snapshot timestamp;
[0012] Traversing the backup copy list, and performing a preset operation on the stand-alone database corresponding to each backup copy in parallel, wherein the preset operation includes querying target data using the snapshot timestamp, and the target data is data that satisfies global transaction consistency;
[0013] The target data returned from each backup copy is summarized and returned to the user.
[0014] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising: at least one processor; and
[0015] a memory communicatively connected to the at least one processor; wherein,
[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed database query method supporting multi-copy consistent reading as described in any embodiment of the present invention.
[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed database query method supporting multi-copy consistent reading as described in any embodiment of the present invention when executed.
[0018] The technical solution of the embodiment of the present invention calculates the backup copy list and snapshot timestamp based on the shard storage list, distributed read consistency level and backup copy selection strategy. The distributed database entry uses this snapshot timestamp to query the latest data version that meets the global transaction consistency on each backup copy in the backup copy list. There is no need to wait for the data of each backup copy to be synchronized, thereby shortening the overall query delay.
[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 An architectural diagram of a multi-version concurrency control method based on a global clock service provided by the prior art;
[0022] Figure 2 A schematic diagram of the structure of a distributed database query system supporting multi-copy consistent reading provided in the first embodiment of the present invention;
[0023] Figure 3 A schematic diagram of a process of obtaining the maximum synchronization timestamp of a backup copy in a distributed database query method supporting multi-copy consistent reading provided by an exemplary embodiment of the present invention;
[0024] Figure 4 A flow chart of a distributed database query method supporting multi-copy consistent read provided in Embodiment 2 of the present invention;
[0025] Figure 5 A flowchart of a distributed database query method supporting multi-copy consistent reading provided by a specific embodiment of the present invention;
[0026] Figure 6 A schematic diagram of the structure of an electronic device supporting a distributed database query method for multi-copy consistent reading according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only an embodiment of a part of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention. It should be understood that the various steps recorded in the method implementation of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.
[0032] Figure 1 The architecture diagram of a multi-version concurrency control method based on a global clock service provided by the prior art is as follows: Figure 1 As shown, the main architecture is as follows:
[0033] Gate: The entrance to the distributed database, which is mainly responsible for processing client requests and distributing them to the master of the stand-alone database in each shard storage, and is also responsible for controlling the execution process of distributed transactions;
[0034] Sharded storage: Data is distributed in different shard storages according to a certain sharding strategy. In order to ensure high availability, the sharded storage contains multiple stand-alone database copies, including one master and multiple backups. Gate reads and writes data from the master by default. All backups synchronize with the master data to ensure the final consistency of the data. The method of data synchronization is not limited. Generally speaking, in order not to affect the speed of the master write, an asynchronous synchronization method is adopted (after the master copy writes the data successfully, it returns to Gate to write successfully, and there is no need to wait for the data to be synchronized to the backup copy). That is, there will be a certain delay in data synchronization. At a certain point in time, the data on the backup copy may not be the latest updated data on the master copy;
[0035] TSO Server: global clock service, responsible for distributing globally unique increasing timestamps;
[0036] Snapshot-TS (sts): When a distributed transaction starts, the Gate is obtained from the TSO Server. When each stand-alone database transaction is started, the sts is issued. That is, each stand-alone database transaction belonging to this distributed transaction has the same sts. Each stand-alone database transaction uses the same sts to query data to meet the ACID consistency of the distributed transaction.
[0037] Commit-TS (cts): Before a distributed transaction is committed, the Gate is obtained from the TSO Server. When each stand-alone database transaction is submitted, cts is issued, that is, each stand-alone database transaction belonging to this distributed transaction has the same cts to meet the ACID consistency of the distributed transaction.
[0038] Stand-alone database: Using global multi-version concurrency control technology, transactions are write operations that modify the latest version of data and generate historical versions of the data. Each version of the data adds persistence to the cts column, which represents the cts of the distributed transaction that updates this data version. When a read operation occurs, according to the snapshot (sts) obtained at the beginning of the transaction where the read operation is located, starting from the latest version, the historical version data is traversed in reverse order to find the first version of the data item that can be read. Stand-alone database transactions belonging to distributed transactions determine whether the data is visible based on sts (a certain version of data can be read, which is generally referred to as a certain version of data visibility), so as to ensure global transaction read consistency.
[0039] When the primary copy of a stand-alone database in each shard storage has a large number of read requests and a high read pressure, the read traffic can be diverted through the read-write separation function to reduce the read pressure on the storage layer. This supports selecting one of the multiple backup copies of a stand-alone database to read data.
[0040] Embodiment 1
[0041] Figure 2 A structural diagram of a distributed database query system supporting multi-copy consistent reading provided in Example 1 of the present invention, the system can be applied to data query of a distributed database, and the system can be implemented by software.
[0042] like Figure 2 As shown, a distributed database query system supporting multi-copy consistent reading is provided in Embodiment 1 of the present invention, including:
[0043] A multi-copy consistency read module 110, a global transaction read consistency module 120, and a multi-copy read management module 130;
[0044] A multi-copy consistent read module 110, configured to provide at least one non-strict distributed read consistency level;
[0045] A global transaction read consistency module 120, used to ensure global transaction read consistency;
[0046] The multi-copy read management module 130 is used to select a final copy from multiple copies of the stand-alone database stored in each shard to read data after the global transaction read consistency module 120 ensures the global transaction read consistency, based on the shard storage list, the non-strict distributed read consistency level and the backup copy selection strategy. The final backup copy meets the requirements of global transaction read consistency and data version close to the latest.
[0047] In this embodiment, the distributed storage system usually provides different consistency levels. It is understandable that a stricter consistency model means a loss in performance (latency), availability or scalability (the number of nodes that can provide services, etc.); a strict distributed read consistency level requires reading the latest version of data from the backup copy, that is, as long as the transaction has been committed to the primary copy, the backup copy can read the data version updated by this transaction.
[0048] The multi-copy consistent read module 110 can support the following non-strict distributed read consistency levels:
[0049] Prefix consistency: Try to read the latest version of data to ensure session-level monotonic reading, that is, if a client reads a certain version n of the data, then the version it reads afterwards must be greater than or equal to n. It cannot guarantee that the read operation will read the latest version of the data;
[0050] Eventual consistency: Try to read the latest version of the data. It cannot guarantee monotonic reading at the session level, and it cannot guarantee that the read operation reads the latest version of the data.
[0051] Among them, users can specify the distributed read consistency level of multi-copy reads by setting system variables.
[0052] In this embodiment, no matter which distributed read consistency level of multi-copy read is selected, global transaction read consistency is guaranteed by default. Global transaction read consistency includes: data synchronization of the standby replica of the stand-alone database will synchronize the commit timestamp of the transaction; the standby replica of the stand-alone database maintains the maximum commit timestamp of the synchronized transaction and provides a query interface; the entrance of the distributed database regularly executes heartbeat distributed transactions.
[0053] Among them, data synchronization is based on transactions. When the standby replica synchronizes the transaction data of the primary replica, it also synchronizes the transaction commit timestamp cts. That is, the transaction synchronized by the standby replica of the stand-alone database has the same cts as the transaction corresponding to the primary replica.
[0054] The standby replica of the stand-alone database maintains the maximum synchronization timestamp, that is, the maximum timestamp of the transactions that have been synchronized. Specifically, each time a transaction is synchronized, if the timestamp of this transaction is greater than the maximum synchronization timestamp, the maximum synchronization timestamp is updated to the timestamp of this transaction. The standby replica of the stand-alone database provides a query interface for the maximum synchronization timestamp.
[0055] Among them, the heartbeat distributed transaction is a distributed transaction triggered inside the Gate, not a transaction initiated by the user. The heartbeat distributed transaction is executed periodically, and it will start and commit a single-machine transaction on the primary copy of the single-machine database stored in all shards.
[0056] In this embodiment, a multi-copy read management module 130 is newly added inside the entrance of the distributed database to manage the consistency of multi-copy reads. The multi-copy read management module 130 can be used to provide a single-machine database backup copy list and snapshot timestamp for multi-copy reads based on the backup copy selection strategy and the distributed read consistency level, while ensuring global transaction consistency, providing efficient query and trying to read the latest data version.
[0057] Further, the multi-copy read management module 130 is used to: calculate the backup copy list and the snapshot timestamp based on the shard storage list, the distributed read consistency level and the backup copy selection strategy, so as to query the data satisfying the global transaction consistency and having the data version close to the latest on each backup copy in the backup copy list by using the snapshot timestamp through the entrance of the distributed database;
[0058] The backup copy selection strategy includes a load balancing strategy and a data version strategy, and the backup copy selection strategy for multi-copy reading is indicated by the second system variable.
[0059] In this embodiment, the following backup copy selection strategies are supported to select a backup copy from the shard storage to read data:
[0060] Load balancing strategy: Use polling to select the next backup replica to read data. When the read traffic is very large, you can choose this strategy to divert the read traffic to multiple backup replicas to improve the overall read performance of the distributed database.
[0061] Data version strategy: Each time, the backup replica with the fastest data synchronization is selected to read data. When you expect to read newer data, you can select this strategy to read the latest version of data synchronized from all backup replicas. The data version strategy can provide the latest data version from all backup replicas.
[0062] The user can specify the backup copy selection policy for multi-copy read by setting the second system variable replica_read_node_select_policy. When the value of the second system variable is 0, the load balancing policy is selected. When the value of the second system variable is 1, the data version policy is selected.
[0063] A distributed database query system supporting multi-copy consistent reading is provided in the first embodiment of the present invention. First, the multi-copy consistent reading module is used to provide at least one non-strict distributed reading consistency level; then the global transaction read consistency module is used to ensure global transaction read consistency; finally, the multi-copy read management module is used for the global transaction read consistency module to ensure global transaction read consistency, and then, based on the shard storage list, the non-strict distributed read consistency level and the backup copy selection strategy, a final copy is selected from multiple copies of the stand-alone database stored in each shard to read data, and the final backup copy meets the requirements of global transaction read consistency and data version close to the latest. While ensuring global transaction read consistency, the above system supports selecting a copy from multiple copies of the stand-alone database stored in each shard to read data, diverts read traffic, reduces the read pressure of the primary copy of the stand-alone database in the shard storage, and improves the overall performance of the distributed database.
[0064] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.
[0065] In one embodiment, the at least one non-strict distributed read consistency level is indicated by a first system variable, including:
[0066] When the first system variable takes the first value, it indicates to read data from the primary copy;
[0067] When the value of the first system variable is the second value, it indicates that multi-copy reading is used, and the non-strict distributed reading consistency level is prefix consistency, and the prefix consistency supports session-level monotonic reading;
[0068] When the first system variable takes the third value, it indicates that multi-copy reading is used, and the non-strict distributed reading consistency level is eventual consistency, and the eventual consistency does not support session-level monotonic reading;
[0069] The first value, the second value and the third value are different values.
[0070] Exemplarily, the user specifies the distributed read consistency level of multi-replica reads by setting the first system variable replica_read_consistency. The supported values are as follows:
[0071] 0: Do not use multi-copy reading, that is, read data from the primary copy. The default value is 0;
[0072] 1: Use multi-replica reads, and the distributed read consistency level is prefix consistency;
[0073] 2: Use multi-replica reads, and the distributed read consistency level is eventual consistency.
[0074] In one embodiment, the process of the multi-copy read management module 130 acquiring the backup copy list and the snapshot timestamp includes:
[0075] The multi-copy read management module receives the shard storage list and the last used snapshot timestamp, initializes the backup copy list to be returned to be empty, and initializes the snapshot timestamp to be returned to be 0;
[0076] Traverse the shard storage list and execute the following process for the current shard storage:
[0077] Initialize the target timestamp to 0 and the target backup copy to be empty; execute different processes to update the target backup copy and the target timestamp according to the value of the second system variable or the value of the first system variable; use the last backup copy for multi-copy reading in the current shard storage as the updated target backup copy; add the updated target backup copy to the list of backup copies to be returned;
[0078] If the snapshot timestamp to be returned is 0, the snapshot timestamp to be returned is updated to the target timestamp; otherwise, if the target timestamp is smaller than the snapshot timestamp to be returned, the snapshot timestamp to be returned is updated to the target timestamp.
[0079] Further, according to the value of the second system variable or the value of the first system variable, different processes are executed to update the target backup copy and the target timestamp, including:
[0080] If the second system variable is the first value, polling and selecting the next standby copy that meets the target condition from the standby copy list stored in the current shard as the target standby copy, wherein the standby copy list is obtained from the standby copy hash table;
[0081] If the second system variable is the second value, the list of backup copies stored in the current shard is traversed, and the following process is performed for the current backup copy: the maximum synchronization timestamp of the current backup copy is obtained from the maximum synchronization timestamp hash table of the backup copies, and if the maximum synchronization timestamp is greater than the target timestamp, the target timestamp is updated to the maximum synchronization timestamp, and the current backup copy is used as the target backup copy;
[0082] If the first system variable is the second value and the last used snapshot timestamp is greater than the target timestamp, the target timestamp is updated to 0, the target backup copy is updated to empty, and the process of re-executing the second system variable to the second value is executed after waiting for the valid time.
[0083] Further, the next standby copy that meets the target condition is selected from the standby copy list stored in the current shard as the target standby copy, including:
[0084] If the first system variable takes the second value, the target condition is set to that the maximum synchronization timestamp of the standby replica is greater than or equal to the last used snapshot timestamp, otherwise, the target condition is set to null;
[0085] Get the last backup copy for multi-copy reading in the current shard storage, determine the next backup copy of the backup copy, and start to cycle the backup copy list of the current shard storage from the next backup copy, and perform the following operations on the current backup copy that is cycled:
[0086] Get the maximum synchronization timestamp of the current standby replica from the maximum synchronization timestamp hash table of the standby replica. If the current standby replica meets the target condition, assign the current standby replica to the target standby replica and assign the maximum synchronization timestamp of the current standby replica to the target timestamp.
[0087] Repeat the above operation until the backup copy list is completed.
[0088] Exemplarily, the main process of obtaining the backup copy list and snapshot timestamp is as follows:
[0089] a. The multi-copy read management module receives the shard storage list and the last used snapshot timestamp;
[0090] b. Initialize the list of backup copies to be returned to be empty, and initialize the timestamp of the snapshot to be returned to be 0;
[0091] c. Traverse the shard storage list. For the current shard storage, execute the following process:
[0092] a) Initialize the target timestamp to 0 and the target backup copy to be empty;
[0093] b) If the value of the second system variable is 0, the next backup copy that meets the target condition is selected from the list of backup copies stored in the current shard as the target backup copy:
[0094] i. Get the index last_index of the standby copy selected in the last copy read of the current shard storage. The index indicates the position of the standby copy in the standby copy list. The corresponding standby copy can be found in the standby copy list through the index. For example, if the standby copy is ranked fourth in the standby copy list, the index of the standby copy is 4; if this is the first standby copy read of the current shard storage, last_index is -1; specifically, last_index is stored in the hash table last_index_map, the last_index of the current shard storage shard is last_index_map[shard], and the current index cur_index is set to last_index+1;
[0095] ii. If the value of the first system variable is the second value: set the target condition to be that the maximum synchronization timestamp of the backup replica is greater than or equal to the last used snapshot timestamp; otherwise, the target condition is empty;
[0096] iii. Find the current backup copy in the backup copy list stored in the current shard through cur_index, and obtain the maximum synchronization timestamp of the current backup copy from the maximum synchronization timestamp hash table of the backup copy. If the current backup copy meets the target condition (if the target condition is empty, then the current backup copy meets the target condition), assign the current backup copy as the target backup copy and go to step e); otherwise, update cur_index to last_index+1 and repeat the current step;
[0097] c) If the second system variable is 1, obtain all backup copies of the shard storage, traverse the backup copy list of the current shard storage, and execute the following process for the current backup copy:
[0098] i. Obtain the maximum synchronization timestamp of the current standby replica from the maximum synchronization timestamp hash table of the standby replica;
[0099] ii. If the maximum synchronization timestamp is greater than the target timestamp, the target timestamp is updated to the maximum synchronization timestamp, and the target backup copy is updated to be empty;
[0100] d) If the first system variable is 1 and the last used snapshot timestamp is greater than the target timestamp, the target timestamp is updated to 0, the target backup copy is empty, and step c) is executed again after waiting for the valid time, otherwise proceed to the next step;
[0101] e) Update the index of the backup copy selected in the last copy read of the current shard storage to the index i of the target backup copy in the backup copy list of the current shard storage. Specifically, last_index_map[shard]=i;
[0102] f), adding the updated target backup copy to the list of backup copies to be returned;
[0103] g) If the timestamp of the snapshot to be returned is 0, the timestamp of the snapshot to be returned is updated to the target timestamp; otherwise, if the target timestamp is less than the timestamp of the snapshot to be returned, the timestamp of the snapshot to be returned is updated to the target timestamp;
[0104] d. Return the list of backup copies to be returned and the timestamp of the snapshot to be returned.
[0105] Furthermore, the maximum synchronization timestamp hash table of the backup copy is stored in the form of a key-value pair, where the key is the unique identifier of the backup copy, and the value is the maximum synchronization timestamp and update time of the backup copy, and the update time is the time when the maximum synchronization timestamp is obtained from the stand-alone database corresponding to the backup copy.
[0106] Among them, the maximum synchronization timestamp maintained by the maximum synchronization timestamp hash table of the standby replica has a validity period. If the maximum synchronization timestamp of a standby replica expires, it is necessary to obtain the earliest maximum synchronization timestamp from the stand-alone database corresponding to the standby replica and update it to the maximum synchronization timestamp hash table of the standby replica; the validity period can be changed through system configuration to reduce the pressure on the standby replica. During the validity period, the standby replica will only be accessed once at most to obtain its maximum synchronization timestamp.
[0107] Furthermore, the backup copy hash table is a hash table of backup copies of all stand-alone databases that maintain all shard storages, with the key being the shard storage and the value being the backup copy list. The backup copy list stores the unique identifiers of each backup copy belonging to the shard storage.
[0108] Among them, the backup copy hash table is initialized when the distributed database is started, and can be automatically updated when the topology of the distributed database changes, and the specific method is not limited; the change in the topology of the distributed database refers to a change in the number of shard storages, or a change in the number of backup copies of a shard storage, etc.
[0109] Furthermore, the process of obtaining a maximum synchronization timestamp of a standby replica from the maximum synchronization timestamp hash table of the standby replica includes:
[0110] If the standby copy is included in the standby copy's maximum synchronization timestamp hash table, the maximum synchronization timestamp and update time of the standby copy are obtained from a standby copy in the standby copy's maximum synchronization timestamp hash table; if the update time has expired, the latest maximum synchronization timestamp of the standby copy is obtained from the stand-alone database corresponding to the standby copy hash table, the latest maximum synchronization timestamp is updated to the maximum synchronization timestamp hash table, and the latest maximum synchronization timestamp is returned; otherwise, the maximum synchronization timestamp is returned; if the standby copy is not included in the standby copy's maximum synchronization timestamp hash table, the latest maximum synchronization timestamp of the standby copy is obtained from the stand-alone database corresponding to the standby copy hash table, the latest maximum synchronization timestamp is updated to the maximum synchronization timestamp hash table, and the latest maximum synchronization timestamp is returned.
[0111] For example, Figure 3 A schematic diagram of a process of obtaining the maximum synchronization timestamp of a backup copy in a distributed database query method supporting multi-copy consistent reading provided by an exemplary embodiment of the present invention, such as Figure 3 As shown in the figure, the main process of obtaining the maximum synchronization timestamp of a standby replica from the maximum synchronization timestamp hash table is as follows:
[0112] a. Receive the backup copy. If the backup copy contained in the maximum synchronization timestamp hash table of the backup copy is not empty, that is, max_sync_cts_map[shard_replica] is not empty, then obtain the maximum synchronization timestamp replica_max_sync_cts and update time update_time of the backup copy from max_sync_cts_map[shard_replica]. If update_time has expired, specifically, if the current system time update_time is greater than the validity period valid_time, then go to step c; otherwise, return replica_max_sync_cts;
[0113] b. If max_sync_cts_map[shard_replica] is empty, go to step c;
[0114] c. Get the latest new_max_sync_cts from the stand-alone database corresponding to the standby replica hash table. Specifically, call the maximum synchronization timestamp query interface provided by it, then update new_max_sync_cts and the current system time to max_sync_cts_map[shard_replica], and return replica_max_sync_cts.
[0115] Embodiment 2
[0116] Figure 4 A flow chart of a distributed database query method supporting multi-copy consistent reading provided in Embodiment 2 of the present invention is applicable to situations where data queries are performed on distributed databases. The method can be executed by a distributed database query system supporting multi-copy consistent reading provided in any embodiment of the present invention.
[0117] like Figure 4 As shown, a distributed database query method supporting multi-copy consistent reading provided by Embodiment 2 of the present invention includes the following steps:
[0118] S110: After receiving the structured query statement sent by the user, the distributed database entry parses the structured query statement and determines a shard storage list where the structured query statement to be queried is distributed.
[0119] S120: If multi-copy reading is used, obtain the backup copy list and snapshot timestamp.
[0120] S130, traversing the backup copy list, and performing preset operations on the stand-alone database corresponding to each backup copy in parallel, wherein the preset operations include using the snapshot timestamp to query target data, and the target data is data that satisfies global transaction consistency.
[0121] S140: Aggregate the target data returned from each backup copy and return it to the user.
[0122] The above method is based on the following implementation:
[0123] a. The data synchronization of the standby copy of the stand-alone database will synchronize the timestamp of the transaction;
[0124] b. The backup copy of the stand-alone database maintains the maximum synchronization timestamp and provides a query interface;
[0125] c. The distributed database entry periodically executes heartbeat distributed transactions.
[0126] A distributed database query method supporting multi-copy consistent reading provided by the second embodiment of the present invention, first, after receiving the structured query statement sent by the user, the distributed database entry parses the structured query statement and determines the shard storage list where the structured query statement to be queried is distributed; then, if multi-copy reading is used, the backup copy list and the snapshot timestamp are obtained; then, the backup copy list is traversed, and the preset operation is performed on the stand-alone database corresponding to each backup copy in parallel, and the preset operation includes using the snapshot timestamp to query the target data, and the target data is data that satisfies the global transaction consistency; finally, the target data returned from each backup copy is summarized and returned to the user. The method can quickly obtain the maximum synchronization timestamp of a backup copy stored in a certain shard through the distributed database entry, and at the same time minimize the call of the maximum synchronization timestamp query interface of the backup copy; the method is based on the shard storage list, the distributed read consistency level and the backup copy selection strategy, and calculates the backup copy list and the snapshot timestamp. The distributed database entry uses this snapshot timestamp to query the latest data version that satisfies the global transaction consistency on each backup copy in the backup copy list, without waiting for the synchronization of each backup copy data, thereby shortening the overall query delay.
[0127] The embodiment of the present invention provides a specific implementation method based on the technical solutions of the above embodiments.
[0128] As a specific implementation method of this embodiment, Figure 5 A flowchart of a distributed database query method supporting multi-copy consistent read is provided for a specific embodiment of the present invention, such as Figure 5 As shown in the figure, while ensuring the read consistency of global transactions, the main process of multi-copy reading is as follows:
[0129] a. After receiving the structured query statement sql sent by the user, the distributed database entry Gate parses the structured query statement to obtain the shard storage collections where the data to be queried is distributed;
[0130] b. If the value of the first system variable replica_read_consistency is 0, multi-replica read is not used; otherwise, execute step c;
[0131] c. Send the shard storage collection and the snapshot timestamp last_sts used by the last multi-copy read of the current session to the multi-copy read management module to obtain the backup replica list replica_list and the snapshot timestamp sts. The current session refers to the thread in which the Gate processes user requests. All SQL statements sent by this user are processed by this thread. If the current SQL statement is the first multi-copy read of the current session, last_sts is 0.
[0132] d. Traverse replica_list and perform the following operations on the stand-alone database corresponding to each backup replica in parallel:
[0133] Use sts to query data. sts is calculated by the multi-copy read management module. It ensures that the max_sync_cts to which the replica data is synchronized is greater than or equal to sts. That is, this query does not need to wait for data synchronization. It directly determines the visibility of the data based on sts and returns the data that satisfies the global transaction consistency of the Gate.
[0134] e. Record last_sts as sts;
[0135] f. Summarize the data returned from each backup copy and return it to the user.
[0136] Embodiment 3
[0137] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0138] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0139] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0140] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a distributed database query method that supports multi-copy consistency reading.
[0141] In some embodiments, a distributed database query method supporting multi-copy consistent reading may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the distributed database query method supporting multi-copy consistent reading described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute a distributed database query method supporting multi-copy consistent reading in any other appropriate manner (e.g., by means of firmware).
[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0143] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0144] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0145] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0146] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0147] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A distributed database query system supporting multi-copy consistent reading, characterized in that: The system includes a multi-copy consistency read module, a global transaction read consistency module and a multi-copy read management module; The multi-copy consistency read module is used to provide at least one non-strict distributed read consistency level; The global transaction read consistency module is used to ensure the global transaction read consistency; The multi-copy read management module is used to select a final copy from multiple copies of the stand-alone database stored in each shard to read data after the global transaction read consistency module ensures the global transaction read consistency, based on the shard storage list, the non-strict distributed read consistency level and the backup copy selection strategy, and the final copy meets the requirements of global transaction read consistency and data version close to the latest; The multi-copy read management module is used for: Based on the shard storage list, the non-strict distributed read consistency level and the backup copy selection strategy, the backup copy list and the snapshot timestamp are calculated, so as to query the data that meets the global transaction consistency and has a data version close to the latest on each backup copy in the backup copy list through the entrance of the distributed database using the snapshot timestamp; The backup copy selection strategy includes a load balancing strategy and a data version strategy, and the backup copy selection strategy for multi-copy reading is indicated by the second system variable; The process of the multi-copy read management module acquiring the backup copy list and the snapshot timestamp includes: The multi-copy read management module receives the shard storage list and the last used snapshot timestamp, initializes the backup copy list to be returned to be empty, and initializes the snapshot timestamp to be returned to be 0; Traverse the shard storage list and execute the following process for the current shard storage: Initialize the target timestamp to 0 and the target backup copy to be empty; execute different processes to update the target backup copy and the target timestamp according to the value of the second system variable or the value of the first system variable; use the last backup copy for multi-copy reading in the current shard storage as the updated target backup copy; Adding the updated target backup copy to the list of backup copies to be returned; If the snapshot timestamp to be returned is 0, then the snapshot timestamp to be returned is updated to the target timestamp; otherwise, if the target timestamp is less than the snapshot timestamp to be returned, then the snapshot timestamp to be returned is updated to the target timestamp; According to the value of the second system variable or the value of the first system variable, different processes are executed to update the target backup copy and the target timestamp, including: If the second system variable is the first value, polling and selecting the next standby copy that meets the target condition from the standby copy list stored in the current shard as the target standby copy, wherein the standby copy list is obtained from the standby copy hash table; If the second system variable is the second value, the list of backup copies stored in the current shard is traversed, and the following process is performed for the current backup copy: the maximum synchronization timestamp of the current backup copy is obtained from the maximum synchronization timestamp hash table of the backup copies, and if the maximum synchronization timestamp is greater than the target timestamp, the target timestamp is updated to the maximum synchronization timestamp, and the current backup copy is used as the target backup copy; If the first system variable is the second value and the last used snapshot timestamp is greater than the target timestamp, the target timestamp is updated to 0, the target backup copy is updated to empty, and the process of re-executing the second system variable to the second value is executed after waiting for the valid time.
2. The system according to claim 1, characterized in that The at least one non-strict distributed read consistency level is indicated by a first system variable, including: When the first system variable takes the first value, it indicates to read data from the primary copy; When the value of the first system variable is the second value, it indicates that multi-copy reading is used, and the non-strict distributed reading consistency level is prefix consistency, and the prefix consistency supports session-level monotonic reading; When the first system variable takes the third value, it indicates that multi-copy reading is used, and the non-strict distributed reading consistency level is eventual consistency, and the eventual consistency does not support session-level monotonic reading; The first value, the second value and the third value are different values.
3. The system according to claim 1, characterized in that The global transaction read consistency includes: The data synchronization of the standby replica of a standalone database will synchronize the commit timestamp of the transaction; The standby copy of the stand-alone database maintains the maximum synchronization timestamp of synchronized transactions and provides a query interface; The entry point of the distributed database periodically executes heartbeat distributed transactions.
4. The system according to claim 1, characterized in that Poll and select the next standby copy that meets the target conditions from the list of standby copies stored in the current shard as the target standby copy, including: If the first system variable takes the second value, the target condition is set to that the maximum synchronization timestamp of the standby replica is greater than or equal to the last used snapshot timestamp, otherwise, the target condition is set to null; Get the last backup copy for multi-copy reading in the current shard storage, determine the next backup copy of the backup copy, and start to cycle the backup copy list of the current shard storage from the next backup copy, and perform the following operations on the current backup copy that is cycled: Get the maximum synchronization timestamp of the current standby replica from the maximum synchronization timestamp hash table of the standby replica. If the current standby replica meets the target condition, assign the current standby replica to the target standby replica and assign the maximum synchronization timestamp of the current standby replica to the target timestamp. Repeat the above operation until the backup copy list is completed.
5. The system according to claim 1, characterized in that The maximum synchronization timestamp hash table of the backup copy is stored in the form of a key-value pair, where the key is the unique identifier of the backup copy, and the value is the maximum synchronization timestamp and update time of the backup copy. The update time is the time when the maximum synchronization timestamp is obtained from the stand-alone database corresponding to the backup copy.
6. The system according to claim 1, characterized in that The process of obtaining the maximum synchronization timestamp of a standby replica from the maximum synchronization timestamp hash table of the standby replica includes: If the standby copy is included in the maximum synchronization timestamp hash table of the standby copy, the maximum synchronization timestamp and update time of the standby copy are obtained from a standby copy in the maximum synchronization timestamp hash table of the standby copy. If the update time has expired, the latest maximum synchronization timestamp of the standby copy is obtained from the stand-alone database corresponding to the standby copy hash table, the latest maximum synchronization timestamp is updated to the maximum synchronization timestamp hash table, and the latest maximum synchronization timestamp is returned; otherwise, the maximum synchronization timestamp is returned; If the backup copy is not included in the maximum synchronization timestamp hash table of the backup copy, obtain the latest maximum synchronization timestamp of the backup copy from the stand-alone database corresponding to the backup copy hash table, update the latest maximum synchronization timestamp to the maximum synchronization timestamp hash table, and return the latest maximum synchronization timestamp.
7. A distributed database query method supporting multi-copy consistent reading, characterized in that: The method comprises: After receiving the structured query statement sent by the user, the distributed database entrance parses the structured query statement and determines the shard storage list where the structured query statement to be queried is distributed; If multi-replica read is used, obtain the backup replica list and snapshot timestamp; Traversing the backup copy list, and performing a preset operation on the stand-alone database corresponding to each backup copy in parallel, wherein the preset operation includes querying target data using the snapshot timestamp, and the target data is data that satisfies global transaction consistency; The target data returned from each backup copy is summarized and returned to the user.
8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed database query method supporting multi-copy consistent reading as described in claim 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed database query method supporting multi-copy consistent reading as described in claim 7 when executed.
