Read control method and device of distributed database, electronic equipment and storage medium

Abstract

The invention relates to a read control method and device for a distributed database, electronic equipment and a storage medium, and the method comprises the steps: receiving a first global transaction identifier of a latest submitted transaction reported by a main storage node in the distributed database, and a second global transaction identifier of a latest replayed transaction reported by a standby storage node; updating the recorded current progress identifier according to the first global transaction identifier and the second global transaction identifier; and in response to a read request from the computing node, determining a target standby storage node of which the synchronization state is consistent with that of the main storage node according to the current progress identifier of each node, so that the computing node executes a read operation based on the target standby storage node. By means of the mode, it is ensured that in the read-write separation scheme, the target standby storage node is determined from the standby storage nodes, strong consistency guarantee of reading data from the standby storage nodes in the distributed environment is achieved, and real-time and reliable data can still be read while services enjoy the expansibility advantage brought by read-write separation.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a read control method, apparatus, electronic device and storage medium for a distributed database. Background Technology

[0002] With the continuous development of database technology, the growth of data volume and the increase in business complexity, read-write separation has been widely used in the field of database technology. Read-write separation technology can improve system throughput and resource utilization.

[0003] However, in existing read / write splitting technologies, after the master node commits a transaction, its data changes are asynchronously transmitted to each backup node via log replication. The backup nodes then replay these logs to update their local data, a process that inevitably involves inherent latency. Therefore, the data that a business reads from a backup node is essentially a snapshot of the master node's data at a certain point in the past, not the current real-time data. Summary of the Invention

[0004] This application provides a read control method, device, electronic device, and storage medium for a distributed database to solve the technical problem that existing read-write separation technologies cannot guarantee the real-time consistency of data read from backup storage nodes.

[0005] In a first aspect, this application provides a read control method for a distributed database, the method comprising: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier; Receive the second global transaction identifier of the latest replayable transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier; In response to a read request from a compute node, a target backup storage node is determined from the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0006] In one possible implementation, determining the target backup storage node from among the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node includes: Based on the current commit progress identifier of the master node and the current replay progress identifier of the backup node, determine whether each backup storage node has reached the same data synchronization state as the master storage node. The backup storage nodes that have reached the same data synchronization state as the primary storage node are identified as the candidate node set; Based on a preset node selection strategy, a backup storage node is selected from the candidate node set as the target backup storage node.

[0007] In one possible implementation, determining whether each backup storage node has reached a data synchronization state consistent with the primary storage node based on the current commit progress identifier of the primary node and the current replay progress identifier of the backup nodes includes: Determine whether the current replay progress indicator of each backup storage node meets the preset synchronization condition relative to the current commit progress indicator of the master node. The preset synchronization condition is used to indicate that the backup storage node has replayed all transactions committed by the master storage node.

[0008] In one possible implementation, when the progress identifier is a globally sequenced transaction identifier, the preset synchronization condition is: the current playback progress identifier of the backup storage node is not less than the current commit progress identifier of the master node.

[0009] In one possible implementation, selecting a backup storage node from the candidate node set as the target backup storage node according to a preset node selection strategy includes: Obtain the real-time load metrics for each backup storage node in the candidate node set; Based on the real-time load metric, the backup storage node with the lowest load is selected from the candidate node set as the target backup storage node.

[0010] In one possible implementation, the compute node performs a read operation based on the target backup storage node in the following manner: Initiate a connection to the target backup storage node and send a read request; Receive response data from the target backup storage node in response to the read request; The response data is then subjected to compliance verification. If the response data passes the compliance verification, the response data is returned to the client.

[0011] In one possible implementation, the method further includes: If the target backup storage node is not determined within the first preset time period after the read request is received, the node reselection process is initiated. The final target storage node is determined through the node reselection process; wherein, the final target storage node is the primary storage node or a backup storage node that meets the data synchronization requirements; The information of the final target storage node is sent to the computing node so that the computing node can perform a read operation based on the target storage node.

[0012] In one possible implementation, determining the final target storage node through the node reselection process includes: The operation of determining the target backup storage node is repeated periodically at preset time intervals; If the target backup storage node is successfully identified before the preset maximum number of retries is reached, then the target backup storage node is identified as the target storage node. If the target backup storage node is still not determined after reaching the preset maximum number of retries, then the primary storage node will be determined as the target storage node.

[0013] Secondly, this application provides a read control device for a distributed database, the device comprising: The commit progress identifier maintenance module is used to receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier. The replay progress identifier maintenance module is used to receive the second global transaction identifier of the latest replayed transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier. The target backup storage node determination module is used to respond to a read request from a compute node, and determine a target backup storage node from the backup storage nodes according to the current commit progress identifier of the master node and the current replay progress identifier of the backup node corresponding to each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0014] In one possible implementation, the target backup storage node determination module includes: The status determination unit is used to determine whether each of the backup storage nodes has reached the same data synchronization state as the main storage node based on the current commit progress identifier of the master node and the current replay progress identifier of the backup node. The candidate node determination unit is used to determine the backup storage nodes that have reached the same data synchronization state as the primary storage node as the candidate node set. The target backup storage node determination unit is used to select a backup storage node from the candidate node set as the target backup storage node according to a preset node selection strategy.

[0015] In one possible implementation, the state determination unit includes: The synchronization condition judgment subunit is used to determine whether the current playback progress indicator of each backup storage node meets the preset synchronization condition relative to the current commit progress indicator of the master node. The preset synchronization condition is used to indicate that the backup storage node has played back all transactions committed by the master storage node.

[0016] In one possible implementation, when the progress identifier is a globally ordered transaction identifier, the preset synchronization condition in the synchronization condition judgment subunit is: the current playback progress identifier of the backup storage node is not less than the current commit progress identifier of the master node.

[0017] In one possible implementation, the target backup storage node determination unit is specifically used for: Obtain the real-time load metrics for each backup storage node in the candidate node set; Based on the real-time load metric, the backup storage node with the lowest load is selected from the candidate node set as the target backup storage node.

[0018] In one possible implementation, the computing node of the target backup storage node determination module performs a read operation based on the target backup storage node in the following manner: Initiate a connection to the target backup storage node and send a read request; Receive response data from the target backup storage node in response to the read request; The response data is then subjected to compliance verification. If the response data passes the compliance verification, the response data is returned to the client.

[0019] In one possible implementation, the device further includes: The reselection startup module is used to initiate a node reselection process if a target backup storage node is not determined within a first preset time period after receiving the read request. The target storage node determination module is used to determine the final target storage node through the node reselection process; wherein, the final target storage node is the primary storage node or a backup storage node that meets the data synchronization status requirements; The compute node execution module is used to send the information of the final target storage node to the compute node, so that the compute node can perform a read operation based on the target storage node.

[0020] In one possible implementation, the target storage node determination module is specifically used for: The operation of determining the target backup storage node is repeated periodically at preset time intervals; If the target backup storage node is successfully identified before the preset maximum number of retries is reached, then the target backup storage node is identified as the target storage node. If the target backup storage node is still not determined after reaching the preset maximum number of retries, then the primary storage node will be determined as the target storage node.

[0021] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a read control program for a distributed database stored in the memory, to implement the read control method for the distributed database described in any one of the first aspects.

[0022] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the read control method for the distributed database described in any one aspect.

[0023] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application receives the first global transaction identifier of the latest committed transaction reported by the primary storage node in the distributed database, and the second global transaction identifier of the latest replayed transaction reported by the backup storage node, and updates the recorded current progress identifier according to the first and second global transaction identifiers; in response to a read request from a compute node, a target backup storage node with a synchronization state consistent with the primary storage node is determined according to the current progress identifier of each node, so that the compute node performs a read operation based on the target backup storage node. This ensures that in the read-write separation scheme, the data read from the backup storage node always maintains a high degree of consistency with the primary storage node, achieving strong consistency guarantee for reading data from the backup storage node in a distributed environment. This allows services to enjoy the scalability advantages brought by read-write separation while still being able to read real-time and reliable data. Furthermore, by identifying the target backup storage node from among the various backup storage nodes, read traffic can be safely and evenly distributed to multiple different backup storage nodes. This enables intelligent and balanced distribution of read request traffic, significantly improving the read throughput of the entire database cluster and reducing single-point load pressure. As a result, it provides stable and efficient technical support for online business scenarios with high concurrency and strong consistency requirements. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0027] Figure 1 A flowchart illustrating an embodiment of a read control method for a distributed database provided in this application. Figure 2 A flowchart illustrating an embodiment of another read control method for a distributed database provided in this application; Figure 3 A flowchart illustrating an embodiment of another read control method for a distributed database provided in this application; Figure 4 A structural block diagram of a read control device for a distributed database provided in this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0030] To address the technical challenge of existing read / write splitting technologies failing to guarantee real-time consistency of data read from backup storage nodes, this application provides a read control method, apparatus, electronic device, and storage medium for a distributed database. By ensuring that data read from backup storage nodes remains highly consistent with the primary storage node in a read / write splitting scheme, strong consistency guarantees are achieved for data read from backup storage nodes in a distributed environment. This allows businesses to enjoy the scalability advantages of read / write splitting while still accessing real-time and reliable data. Furthermore, by identifying the target backup storage node from among the various backup storage nodes, read traffic can be securely and evenly distributed across multiple different backup storage nodes. This intelligently and evenly distributes read request traffic, significantly improving the read throughput of the entire database cluster and reducing single-point load pressure, thus providing stable and efficient technical support for online business scenarios with high concurrency and strong consistency requirements.

[0031] Figure 1 A flowchart illustrating an embodiment of a read control method for a distributed database provided in this application includes the following steps: Step 101: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier.

[0032] The aforementioned distributed database can be understood as one where data is not stored on a single server, but rather distributed across primary storage nodes and multiple backup storage nodes. One primary storage node corresponds to one primary server, while multiple backup storage nodes correspond to multiple standby servers. Compute nodes, on the other hand, are dedicated to handling user queries or operation requests; they do not store data but are responsible for scheduling queries.

[0033] The primary storage node refers to the node in a distributed database responsible for handling write operations (insert, update, delete). The latest committed transaction can be understood as the most recent data modification operation that has been successfully completed and committed on the primary storage node.

[0034] The first global transaction identifier is a unique identifier assigned to each database operation transaction. It is globally unique and increments sequentially according to execution order. It is used to mark the execution order of transactions and whether a transaction has been completed, facilitating the determination of whether data on the primary and backup storage nodes are synchronized.

[0035] The current commit progress indicator of the primary node can refer to the identifier of the latest completed transaction or the latest commit progress information of the primary storage node. It can be understood as a progress bar for the primary storage node, indicating where the primary storage node has committed to. For example, the current commit progress indicator of the primary node can be represented as T100, which means that the data of the primary storage node has now been updated to T100.

[0036] In one embodiment, after successfully committing a transaction, the primary storage node will proactively, or through polling by the coordinator, send the first global transaction identifier of its latest committed transaction to the coordinator. Upon receiving the first global transaction identifier, the coordinator updates the first global transaction identifier to the current commit progress identifier of the primary node in the current record.

[0037] For example, the aforementioned first global transaction identifier can be a GTID (Global Transaction Identifier). Specifically, in the entire database cluster, each successfully committed transaction has a unique GTID, and these GTIDs are not duplicated. Furthermore, GTIDs have monotonically increasing characteristics and sequentiality; for a given transaction source, the generated GTIDs are typically single-point increasing. The coordinator in the above embodiment can be a global transaction management module, which can be a core, centralized control and coordination component. It is responsible for maintaining data state consistency information from a global perspective and providing critical routing decisions for read and write requests, serving as the central brain for implementing cross-node, strongly consistent read functionality.

[0038] For example, the primary storage node has just successfully executed the transaction where user B recharges 500 yuan. The GTID of this transaction is T100 (previously it was T99). The primary storage node reports the first global transaction identifier = T100 to the global transaction management module. The global transaction management module updates the current commit progress identifier of the primary storage node, which was originally recorded as T99, to T100. The current commit progress identifier of the primary storage node is now T100.

[0039] Step 102: Receive the second global transaction identifier of the latest replayed transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier.

[0040] A standby storage node is a replica node that copies data from the primary storage node. It is used to share read requests and is also known as a backup machine. Essentially, it is dedicated to synchronizing data with the primary storage node and handling read operations (queries). When the primary storage node fails, it can switch to become a standby storage node to share the read pressure on the primary storage node and achieve disaster recovery backup.

[0041] The latest replayed transaction refers to the last transaction successfully replayed on the standby storage node. Replay refers to the process of applying the host's logs to the local database. Specifically, when the standby storage node successfully receives and synchronizes the transaction from the primary storage node, it essentially replicates the primary storage node's operations. In other words, the standby storage node can only maintain data synchronization with the primary storage node after replaying the transaction.

[0042] The second global transaction identifier can refer to the GTID of the latest replayed transaction reported by the storage node.

[0043] The current replay progress indicator of the backup node can refer to the latest transaction position that each backup storage node has been replayed to, as recorded by the global transaction management module, and can reflect the current data synchronization progress of the backup storage node.

[0044] In one embodiment, after the primary storage node completes a transaction, it synchronizes the transaction log to all backup storage nodes. After receiving the log, each backup storage node performs a replay operation (replicating the transaction of the primary storage node). After the replay is completed, it marks and records the current second global transaction identifier of the backup storage node. Then, each backup storage node reports its latest replayed transaction's second global transaction identifier to the global transaction management module at a fixed frequency. The global transaction management node updates the current replay progress identifier of the backup node according to the second global transaction identifier.

[0045] For example, the primary storage node synchronizes the transaction log of T100 to backup A (backup storage node A) and backup B (backup storage node B). Backup B has a fast network and good performance, completing the replay of T100 within 1 second, and reports the second global transaction identifier = T100 to the global transaction management module. The global transaction management module updates the replay progress identifier of backup B to T100. Backup A has a slightly slow network and has only completed the replay of T98, still processing T99 and T100, so it reports the second global transaction identifier = T98. The global transaction management module retains the current replay progress identifier of backup A as T98. At this time, backup B is synchronized to T100 (indicating that the data of backup B and the primary storage node are consistent), and backup A is synchronized to T98 (indicating that backup A is 2 versions behind the primary storage node, and at this time, the first global transaction identifier of the committed transaction corresponding to the primary storage node has been updated to T100).

[0046] Step 103: In response to a read request from a compute node, determine the target backup storage node from among the backup storage nodes based on the current commit progress flag of the master node and the current replay progress flag of each backup storage node, so that the compute node can perform a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0047] Compute nodes can be nodes used to receive client SQL requests, parse, optimize, and route them. They can also request from the global transaction management node which storage node to use for read operations. Essentially, they can be understood as a middleware layer that connects to users, receiving user query requests (such as querying orders from May 2024), finding a suitable backup storage node to execute the query, and finally returning the results to the user.

[0048] A read request can refer to an SQL query request initiated by the client. The target standby storage node can refer to the standby storage node that is ultimately selected to execute the read request.

[0049] The data synchronization status has been aligned with the primary storage node, meaning that the data on the target standby storage node already includes all committed transactions on the primary storage node up to a certain point in time. In other words, at that moment, the data on the standby and primary storage nodes are in sync. Figure 1 To ensure that the current replay progress flag of the backup storage node is not less than the current commit progress flag of the primary node.

[0050] In one embodiment, the compute node receives a user's read request and applies to the global transaction management module for a target backup storage node. The global transaction management module determines the target backup storage node from multiple backup storage nodes based on the relationship between the current commit progress flag of the master node and the current replay progress flags of all backup nodes. The compute node then sends a query request to the target backup storage node and returns the data results obtained from the query to the user.

[0051] For example, suppose user C wants to check user B's account balance and initiates a read request through the compute node. The query shows that the current progress indicator of the primary node is T100, backup A is T98, and backup B is T100. The global transaction management module determines that the current replay progress indicator of backup B is consistent with the current committed progress indicator of the primary node (both are T100), so backup B is selected as the target storage node. The compute node queries backup B, and because backup B has already replayed T100 (the transaction where user B recharged 500 yuan), it returns a balance of 500 yuan. If backup A (progress T98) had been selected, the returned balance would have been the balance before user B recharged (e.g., 0 yuan), which is old data and does not meet the requirements.

[0052] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application receives the first global transaction identifier of the latest committed transaction reported by the primary storage node in the distributed database, and the second global transaction identifier of the latest replayed transaction reported by the backup storage node, and updates the recorded current progress identifier according to the first and second global transaction identifiers; in response to a read request from a compute node, a target backup storage node with a synchronization state consistent with the primary storage node is determined according to the current progress identifier of each node, so that the compute node performs a read operation based on the target backup storage node. This ensures that in the read-write separation scheme, the data read from the backup storage node is always highly consistent with the primary storage node, achieving strong consistency guarantee for data read from the backup in a distributed environment. This allows services to enjoy the scalability advantages brought by read-write separation while still being able to read real-time and reliable data. Furthermore, by identifying the target backup storage node from among the various backup storage nodes, read traffic can be safely and evenly distributed to multiple different backup storage nodes. This enables intelligent and balanced distribution of read request traffic, significantly improving the read throughput of the entire database cluster and reducing single-point load pressure. As a result, it provides stable and efficient technical support for online business scenarios with high concurrency and strong consistency requirements.

[0053] Figure 2 A flowchart illustrating an embodiment of another read control method for a distributed database provided in this application is shown below. Figure 1 Based on the illustrated process, this section mainly describes how to determine the target backup storage node from multiple backup storage nodes, including the following steps: Step 201: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier.

[0054] Step 202: Receive the second global transaction identifier of the latest replayable transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier.

[0055] For steps 201-202 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0056] Step 203: In response to a read request from a compute node, determine whether each backup storage node has reached the same data synchronization state as the primary storage node based on the current commit progress flag of the primary node and the current replay progress flag of the backup node.

[0057] Step 204: Determine the backup storage nodes that have reached the same data synchronization state as the primary storage node as the candidate node set.

[0058] The following is a unified explanation of steps 203-204 above: Data synchronization consistency means that at a certain point in time, the set of data changes (the set of replayed transactions) owned by the backup storage node completely covers the set of transactions committed by the primary storage node.

[0059] In one embodiment, in response to a read request from a compute node, it is determined whether the current replay progress indicator of each backup storage node meets a preset synchronization condition relative to the current commit progress indicator of the master node. The preset synchronization condition is used to indicate that the backup storage node has replayed all transactions committed by the master storage node.

[0060] Preset synchronization conditions can refer to logical rules used to determine whether the backup storage node has contained all committed transactions of the primary storage node. For example, when the progress identifier is a globally ordered transaction identifier, the preset synchronization condition is: the current replay progress identifier of the backup storage node is not less than the current commit progress identifier of the primary node.

[0061] The candidate node set can refer to a temporary set of nodes containing the identifiers of all backup storage nodes that have passed the data synchronization state consistency check. That is, the data synchronization state of the candidate nodes in this set is consistent with the data synchronization state of the primary storage node.

[0062] Specifically, when a compute node receives a user read request, it applies to the global transaction management module for a target backup storage node. The global transaction management module, based on the data shard identifier in the read request, finds the current commit progress identifier of the primary storage node corresponding to the data shard and the current replay progress identifier of the backup storage nodes of multiple backup storage nodes, verifies whether each backup storage node meets the preset synchronization conditions, and determines all backup storage nodes that meet the preset synchronization conditions as a candidate node set.

[0063] For example, suppose a user needs to query order information, and the data shard corresponding to this order data is represented as order_shard_001. After receiving the query request, the compute node will consult the global transaction management module to determine which backup storage node of this shard should be read from. At this time, the latest committed transaction identifier of the primary storage node of this shard (such as Master_DN_01) is T105, while the synchronization progress of the three backup storage nodes is different: for example, Slave_DN_A has been replayed to T104, Slave_DN_B has been replayed to T105, and Slave_DN_C has only been replayed to T103.

[0064] Furthermore, the global transaction management module, based on preset synchronization conditions—that is, the current replay progress indicator of the backup storage node is not less than the current commit progress indicator of the master node—determines whether the data status of each backup storage node has reached consistency with the master storage node. Specifically, the comparison reveals that the current replay progress indicator (T104) of the Slave_DN_A backup storage node is less than T105 of the master storage node, indicating that it lacks the latest transaction T105 and the data is not synchronized. The current replay transaction indicator of Slave_DN_B is T105, indicating that it has included all committed transactions of the master node, and the data status is completely consistent. The current replay progress indicator T103 of the Slave_DN_C backup storage node is less than T105, indicating that it lacks transactions T104 and T105, and the synchronization is lagging. Therefore, through comparison, it is determined that the backup storage node Slave_DN_B meets the preset synchronization conditions and is added to the candidate node set. This is just an example; if other backup storage nodes meet the preset synchronization conditions, they are directly added to the candidate node set. The specific number of candidate backup storage nodes in the above candidate node set is not limited in this application embodiment.

[0065] Step 205: Based on the preset node selection strategy, select a backup storage node from the candidate node set as the target backup storage node, so that the computing node can perform read operations based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the primary storage node.

[0066] The preset node selection strategy can refer to the rules for selecting the final target backup storage node from the candidate node set. For example, the node with the fastest response or the lowest load can be selected from the candidate node set, or the final target backup storage node can be determined according to the round-robin allocation rules of the global management module.

[0067] In one embodiment, the real-time load index of each backup storage node in the candidate node set is obtained; based on the real-time load index, the backup storage node with the lowest load is determined from the candidate node set as the target backup storage node.

[0068] Real-time load metrics can be understood as data on the current busyness of backup storage nodes, such as CPU utilization, memory usage, the number of currently processed queries, and network bandwidth usage (e.g., backup storage node 1 using 30% of network bandwidth, backup storage node 2 using 70% of network bandwidth). Essentially, it is dynamic data that reflects the current busyness and health of the nodes. This is merely an example; other relevant metrics that reflect the load status of nodes can also be used, and this application embodiment does not impose any limitations on this.

[0069] Specifically, the aforementioned real-time load metrics can include: CPU utilization, reflecting the workload of the node's processors; memory usage, reflecting the utilization of the node's memory resources; current active connections, reflecting the number of database connections being served; query queue length, reflecting the number of query tasks waiting to be processed; network bandwidth utilization, reflecting the throughput of the network interface; and disk I / O rate, reflecting the read / write load of the storage device. Furthermore, the global transaction management module can perform a comprehensive evaluation based on the collected load metrics to obtain a comprehensive evaluation result. Alternatively, it can compare individual metrics, calculating a load score for each stored node and selecting the node with the lowest load (i.e., the lowest load score) as the target backup storage node, as this node is the least busy and has the strongest processing capacity. This method effectively improves resource utilization.

[0070] For example, when there are multiple backup storage nodes in the candidate node set that are synchronizing data with the primary storage node, the global transaction management module will select the least busy node based on the real-time load. For instance, backup storage nodes X, Y, and Z may all have their data synchronized to the latest state, but their real-time loads differ: X has 32 connections and 45% CPU utilization; Y has 58 connections and 75% CPU utilization; while Z has only 15 connections and 20% CPU utilization. If the minimum connection count strategy is adopted, the global transaction management module will select the lightest-loaded backup node Z as the target node to execute the query. This directs new read requests to the server with the least current pressure, avoiding overload of individual backup nodes, thereby shortening query response time and improving the overall processing capacity of the cluster.

[0071] In one embodiment, the compute node performs a read operation based on the target backup storage node by: initiating a connection to the target backup storage node and sending a read request; receiving response data from the target backup storage node in response to the read request; performing compliance verification on the response data; and returning the response data to the client if the response data passes the compliance verification.

[0072] Compliance verification refers to the final quality check performed by the compute node on the query results received from the storage node before returning the data to the client. It is the last line of defense for data correctness and can be regarded as a specific implementation of validity check or liveness detection.

[0073] For example, the compute node uses the address of the target backup storage node returned by the global transaction management module to establish a database connection with the target backup storage node and sends an SQL query. The compute node then receives the query result set returned by the target backup storage node and performs compliance verification on the result set, such as checking whether the result data format is complete and undamaged; or checking whether the connection with the target backup storage node is still healthy and the session is valid after confirming the query result. If the query result set passes the compliance verification, the compute node encapsulates the result set and returns it to the client. If the query result fails the compliance verification, the compute node returns an error message to the client (such as "Query failed, please retry") and may mark the target node as temporarily unavailable.

[0074] For example, when a compute node receives a user's request to query the latest order amount, and selects the lightest-loaded and data-synchronized backup storage node, Slave_DN_Z, as the target backup storage node through the global transaction management module, the compute node will use the node's address information to establish a database connection with it and send the specific SQL query statement. After Slave_DN_Z executes the query, it returns a result data packet containing the order amount (e.g., 500 yuan) to the compute node. Before forwarding the result to the user client, the compute node performs critical compliance checks: First, it parses the data packet and verifies its CRC checksum to confirm that the data is complete and undamaged during transmission; then, the compute node sends a lightweight, instant heartbeat query to Slave_DN_Z (e.g., using the following database query statement to test the connection: SELECT 1) to verify that the connection and session between the two are still active and valid. When both checks pass successfully, the compute node encapsulates the result of 500 yuan and securely returns it to the user. Conversely, if the compute node discovers that the connection has been unexpectedly broken or the heartbeat query is unresponsive after receiving data, it will immediately return a "Query failed, please try again" message to the user and temporarily mark the backup storage node as abnormal, thereby ensuring that the user does not receive potentially incomplete or invalid data.

[0075] pass Figure 2 The illustrated embodiment describes a smart selection strategy based on real-time load metrics, dynamically routing read requests to the most idle and highest-capacity standby machine. This effectively avoids single-point overheating, achieves load balancing, and significantly shortens query response time while improving the overall throughput of the cluster. Finally, by having compute nodes perform compliance checks before returning results, a final quality defense is added to data delivery. This allows for the timely interception and handling of incomplete results caused by network or node anomalies, further enhancing system robustness and user experience. This enables the read-write separation architecture to truly and securely support core online services with stringent requirements for data real-time performance and system performance.

[0076] Figure 3 A flowchart illustrating another embodiment of a read control method for a distributed database provided in this application is shown below. Figure 1 Based on the illustrated process, this section mainly describes how to initiate a node reselection process to determine the target storage node when the target backup storage node has not yet been determined, including the following steps: Step 301: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier.

[0077] Step 302: Receive the second global transaction identifier of the latest replayed transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier.

[0078] Step 303: In response to a read request from a compute node, a target backup storage node is determined from among the backup storage nodes based on the current commit progress flag of the master node and the current replay progress flag of each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0079] For steps 301-303 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0080] Step 304: If the target backup storage node is not determined within the first preset time after receiving the read request, the node reselection process is initiated.

[0081] The first preset duration can refer to a configurable time window used to define the maximum waiting time for the system to attempt to immediately find a qualified standby storage node. It reflects the business's tolerance for the initial response latency of read requests. For example, it can be set to 50 milliseconds or 100 milliseconds.

[0082] "No target backup storage node identified" can mean that no backup storage node meets the preset synchronization conditions or is not in the same data synchronization state as the primary storage node. In other words, the data synchronization progress of all backup storage nodes lags behind that of the primary node.

[0083] The node reselection process can refer to an automatically triggered background retry and decision-making process. When the first attempt fails, the system does not immediately return an error to the client, but instead initiates this process to continue trying over a longer period of time.

[0084] In one embodiment, the global transaction management module starts a timer immediately upon receiving a read request from a compute node. If no qualified standby machine is selected after the timer reaches a first preset duration (e.g., 50ms), a reselection process is triggered.

[0085] Step 305: Determine the final target storage node through the node reselection process; wherein, the final target storage node is either the primary storage node or a backup storage node that meets the data synchronization status requirements.

[0086] In one embodiment, determining the final target storage node through a node reselection process includes: periodically repeating the operation of determining the target backup storage node at preset time intervals; if the target backup storage node is successfully determined before reaching the preset maximum number of retries, then the target backup storage node is determined as the target storage node; if the target backup storage node is still not determined after reaching the preset maximum number of retries, then the primary storage node is determined as the target storage node.

[0087] The preset time interval refers to the periodic interval at which the global transaction management module performs consistency checks during the reselection process. For example, checking the standby machine progress for updates every 100 milliseconds. This interval should be less than the first preset duration to achieve finer-grained waiting.

[0088] The preset maximum number of retries refers to the maximum number of times the reselection process will perform periodic checks. Together with the preset time interval, it determines the total timeout of the reselection process, which is the preset duration mentioned above (total duration ≈ interval × maximum number of retries). For example, with an interval of 100ms and 50 retries, the total timeout (preset duration) is 5 seconds.

[0089] The final target storage node can refer to the final output of the reselection process. It may be that during the retry process, the progress of a backup storage node catches up and meets the conditions, thus being determined as the target storage node; or it may be that after the retry process ends, the target backup storage node is still not determined, in which case the primary storage node is downgraded to be selected as the final target storage node and the auxiliary computing node performs the read operation.

[0090] For example, when the global transaction management module fails to find a standby storage node for data synchronization on its initial attempt, it initiates a node reselection process. This process automatically rechecks the synchronization progress of each standby node at a preset period (e.g., every 100 milliseconds) and sets a maximum number of retries (e.g., 50 times) as a safety boundary. In a specific scenario, the primary storage node has just completed an important transaction, with its progress marked as T200, while the two standby nodes, due to a brief delay, are stuck at T198 and T199 respectively. The global transaction management module initiates a retry. On the third check (approximately 300 milliseconds later), it finds that standby node B's progress has been updated to T200, satisfying the synchronization condition. The global transaction management module immediately terminates the retry and determines standby node B as the final target storage node. Conversely, if after 50 retries (i.e., after 5 seconds), the two standby nodes still fail to catch up with the primary node's progress, the global transaction management module will execute a degradation strategy, determining the primary node itself as the final target, thereby ensuring that read requests always get a usable execution node.

[0091] Step 306: Send the information of the final target storage node to the compute node so that the compute node can perform a read operation based on the target storage node.

[0092] The final target storage node information may include at least the network identifier of the identified node (which may be a backup storage node or a primary storage node), such as IP address, port number, node ID, etc., for establishing a connection with the compute node.

[0093] In one embodiment, the global transaction management module encapsulates the information of the final target storage node into a standard response message and sends this message to the waiting compute nodes. The message can explicitly indicate the corresponding node type (e.g., backup storage node / primary storage node). Upon receiving the response information, the compute node connects and performs read operations in the same manner, regardless of whether the target storage node is a backup or primary storage node.

[0094] For example, after determining the final target storage node, the global transaction management module encapsulates its network information and sends it to the waiting compute nodes. For instance, if the final target is the standby node B found through reselection, the global transaction management module returns a response message similar to {"Node Type": "Standby", "Address": "192.168.1.102:3306"}. Upon receiving this, the compute node connects to that address, executes an SQL query, and returns the result to the user. In a degraded scenario, the information returned by the global transaction management module indicates that the node type is the primary storage node, and the address is the network identifier of the primary storage node. The compute node handles this in the same way, connecting to the primary storage node and performing a read operation. In this way, regardless of whether the underlying layer successfully selects a standby node through waiting or degrades to a primary storage node after a timeout, it is a consistent and successful processing procedure for the upper-layer business requests and compute nodes, ensuring the eventual availability of the service.

[0095] pass Figure 3 The description of the illustrated embodiment illustrates that when the system experiences a brief delay in all standby data due to network jitter, a surge in writes to the primary node, or other reasons, and cannot immediately meet consistency requirements, it does not simply return a query failure to the client. Instead, it proactively initiates a structured and controllable retry waiting process. By periodically rechecking the standby synchronization progress and continuously attempting within a controllable timeframe (e.g., within a few seconds), the system can fully utilize the brief synchronization window to successfully route most delayed requests to the standby node that eventually catches up. This ensures data consistency while minimizing unnecessary query failures. If the standby latency exceeds a preset tolerance range, the solution executes a clear degradation decision, safely directing read requests to the primary storage node. This sacrifices some read scalability to resolutely guarantee the eventual availability of the core read service. It achieves an intelligent and configurable balance between data consistency, service availability, and response latency, providing crucial technical support for ensuring critical business continuity and a smooth user experience.

[0096] Figure 4 A structural block diagram of a read control device for a distributed database provided in this application, the device comprising: The commit progress identifier maintenance module 41 is used to receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier. The replay progress identifier maintenance module 42 is used to receive the second global transaction identifier of the latest replayable transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier. The target backup storage node determination module 43 is used to respond to a read request from a compute node, and determine a target backup storage node from each of the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0097] In one possible implementation, the target backup storage node determination module 43 includes: The status determination unit is used to determine whether each of the backup storage nodes has reached the same data synchronization state as the main storage node based on the current commit progress identifier of the master node and the current replay progress identifier of the backup node. The candidate node determination unit is used to determine the backup storage nodes that have reached the same data synchronization state as the primary storage node as the candidate node set. The target backup storage node determination unit is used to select a backup storage node from the candidate node set as the target backup storage node according to a preset node selection strategy.

[0098] In one possible implementation, the state determination unit includes: The synchronization condition judgment subunit is used to determine whether the current playback progress indicator of each backup storage node meets the preset synchronization condition relative to the current commit progress indicator of the master node. The preset synchronization condition is used to indicate that the backup storage node has played back all transactions committed by the master storage node.

[0099] In one possible implementation, when the progress identifier is a globally ordered transaction identifier, the preset synchronization condition in the synchronization condition judgment subunit is: the current playback progress identifier of the backup storage node is not less than the current commit progress identifier of the master node.

[0100] In one possible implementation, the target backup storage node determination unit is specifically used for: Obtain the real-time load metrics for each backup storage node in the candidate node set; Based on the real-time load metric, the backup storage node with the lowest load is selected from the candidate node set as the target backup storage node.

[0101] In one possible implementation, the computing node of the target backup storage node determination module performs a read operation based on the target backup storage node in the following manner: Initiate a connection to the target backup storage node and send a read request; Receive response data from the target backup storage node in response to the read request; The response data is then subjected to compliance verification. If the response data passes the compliance verification, the response data is returned to the client.

[0102] In one possible implementation, the device further includes: The reselection startup module is used to initiate a node reselection process if a target backup storage node is not determined within a first preset time period after receiving the read request. The target storage node determination module is used to determine the final target storage node through the node reselection process; wherein, the final target storage node is the primary storage node or a backup storage node that meets the data synchronization status requirements; The compute node execution module is used to send the information of the final target storage node to the compute node, so that the compute node can perform a read operation based on the target storage node.

[0103] In one possible implementation, the target storage node determination module is specifically used for: The operation of determining the target backup storage node is repeated periodically at preset time intervals; If the target backup storage node is successfully identified before the preset maximum number of retries is reached, then the target backup storage node is identified as the target storage node. If the target backup storage node is still not determined after reaching the preset maximum number of retries, then the primary storage node will be determined as the target storage node.

[0104] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the read control method for a distributed database provided in any of the foregoing method embodiments, including: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier; Receive the second global transaction identifier of the latest replayable transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier; In response to a read request from a compute node, a target backup storage node is determined from the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

[0105] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the distributed database read control method provided in any of the foregoing method embodiments.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0109] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A read control method for a distributed database, characterized in that, The method includes: Receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier; Receive the second global transaction identifier of the latest replayable transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier; In response to a read request from a compute node, a target backup storage node is determined from the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

2. The method according to claim 1, characterized in that, The step of determining the target backup storage node from among the backup storage nodes based on the current commit progress identifier of the master node and the current replay progress identifier of each backup storage node includes: Based on the current commit progress identifier of the master node and the current replay progress identifier of the backup node, determine whether each backup storage node has reached the same data synchronization state as the master storage node. The backup storage nodes that have reached the same data synchronization state as the primary storage node are identified as the candidate node set; Based on a preset node selection strategy, a backup storage node is selected from the candidate node set as the target backup storage node.

3. The method according to claim 2, characterized in that, The step of determining whether each backup storage node has reached the same data synchronization state as the primary storage node based on the current commit progress identifier of the primary node and the current replay progress identifier of the backup node includes: Determine whether the current replay progress indicator of each backup storage node meets the preset synchronization condition relative to the current commit progress indicator of the master node. The preset synchronization condition is used to indicate that the backup storage node has replayed all transactions committed by the master storage node.

4. The method according to claim 3, characterized in that, When the progress identifier is a globally ordered transaction identifier, the preset synchronization condition is: the current playback progress identifier of the backup storage node is not less than the current commit progress identifier of the master node.

5. The method according to claim 2, characterized in that, The step of selecting a backup storage node from the candidate node set as the target backup storage node according to a preset node selection strategy includes: Obtain the real-time load metrics for each backup storage node in the candidate node set; Based on the real-time load metric, the backup storage node with the lowest load is selected from the candidate node set as the target backup storage node.

6. The method according to claim 1, characterized in that, The compute node performs read operations based on the target backup storage node in the following manner: Initiate a connection to the target backup storage node and send a read request; Receive response data from the target backup storage node in response to the read request; The response data is then subjected to compliance verification. If the response data passes the compliance verification, the response data is returned to the client.

7. The method according to claim 1, characterized in that, The method further includes: If the target backup storage node is not determined within the first preset time period after the read request is received, the node reselection process is initiated. The final target storage node is determined through the node reselection process; wherein, the final target storage node is the primary storage node or a backup storage node that meets the data synchronization requirements; The information of the final target storage node is sent to the computing node so that the computing node can perform a read operation based on the target storage node.

8. The method according to claim 7, characterized in that, The process of determining the final target storage node through the node reselection procedure includes: The operation of determining the target backup storage node is repeated periodically at preset time intervals; If the target backup storage node is successfully identified before the preset maximum number of retries is reached, then the target backup storage node is identified as the target storage node. If the target backup storage node is still not determined after reaching the preset maximum number of retries, then the primary storage node will be determined as the target storage node.

9. A read control device for a distributed database, characterized in that, The device includes: The commit progress identifier maintenance module is used to receive the first global transaction identifier of the latest committed transaction reported by the master storage node in the distributed database, and update the recorded current commit progress identifier of the master node according to the first global transaction identifier. The replay progress identifier maintenance module is used to receive the second global transaction identifier of the latest replayed transaction reported by the backup storage node in the distributed database, and update the recorded current replay progress identifier of the backup node according to the second global transaction identifier. The target backup storage node determination module is used to respond to a read request from a compute node, and determine a target backup storage node from the backup storage nodes according to the current commit progress identifier of the master node and the current replay progress identifier of the backup node corresponding to each backup storage node, so that the compute node performs a read operation based on the target backup storage node, wherein the data synchronization status of the target backup storage node has been consistent with that of the master storage node.

10. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a read control program for a distributed database stored in the memory, to implement the read control method for a distributed database as described in any one of claims 1-8.

11. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the read control method of the distributed database according to any one of claims 1-8.

Images