High availability with consensus in database system

Abstract

A database management system configured to perform a computer-implemented method is provided. The method involves configuring a first set of nodes to determine an instantaneous state of the database management system, generate system configuration data and provide the system configuration data to a second set of nodes of the database management system, where the configuration data is replicated between the first set of nodes according to a consensus-based replication protocol in order to maintain the system configuration data. The method further involves configuring a second set of nodes to receive the system configuration data and implement a state machine arranged to modify the configuration of the second set of nodes in accordance with the system configuration data, where the second set of nodes is arranged to replicate the user data in accordance with a non-consensus-based replication protocol. The invention provides a non-transitory computer readable storage medium. The non-transitory computer readable storage medium comprises instructions for executing a computer-implemented method.

Description

Background Art Technical Field

[0001] The present disclosure relates to database management systems, and more particularly, to methods and systems for managing data availability in a database management system.

[0002] Related technical notes

[0003] As technology advances, the amount of information stored electronically, the demand for real-time or pseudo-real-time access to that information, and the ability to search, organize, and / or manipulate such information continue to increase. A database management system (sometimes also referred to as a database system, database, data store, and data warehouse) organizes data in a format that facilitates efficient searching, retrieval, and / or manipulation of selected information. A typical database management system allows a user to submit a "query" in a query language or invoke one or more functions to search, organize, retrieve, and / or manipulate information that meets specific criteria.

[0004] Some databases may be transactional, meaning their primary purpose is to record transactions. Such transactions can be considered to be one or more operations performed on data recorded in a log. The log may include a continuous stream of log records, each of which corresponds to a transaction. This can allow transactions to be replayed or undone after an event such as a system crash. Some databases may also or alternatively be analytical, meaning their purpose is to execute queries and generate analytics on the data stored in the database.

[0005] The demands on a database system can vary. To address increased demand, a database system can be scaled up (and down) by increasing (and decreasing) the resources of an existing server by adding (or decreasing) memory or upgrading the CPU. Scaling a database system horizontally increases capacity by adding new nodes or additional resources (e.g., in the form of new machines) to the database system.

[0006] Introducing new machines into a database system, and / or maintaining data consistency across the database system in the event of machine failures, presents many technical challenges. Summary of the Invention

[0007] According to a first aspect of the present disclosure, there is provided a computer-implemented method for operating a database management system, the method comprising: configuring a first group of nodes of the database management system to: determine a transient state of the database management system; generating system configuration data representing a target configuration of the database management system based on the transient state; and providing the system configuration data to a second group of nodes of the database management system, wherein the system configuration data is replicated between the first group of nodes according to a consensus-based replication protocol so as to maintain the system configuration data; and configuring the second group of nodes to: receive the system configuration data from the first group of nodes; and implementing a state machine arranged to modify the configuration of the second group of nodes according to the system configuration data, wherein the second group of nodes is arranged to replicate user data according to a non-consensus-based replication protocol.

[0008] The computer-implemented method disclosed herein is intended to produce a database system that provides high availability of user data throughout the database system, such that user operations can continue without significant downtime in the event of a node failure or other error in the database system. This method utilizes a powerful and intensive backup program to manage the configuration of the database system, while simultaneously using a less powerful, but faster and more flexible program to manage the backup operations used to maintain user data. Specifically, a highly durable consensus-based replication protocol is used for backups and, in some cases, generates configuration data for managing the database system. A non-consensus-based replication protocol is used when performing operations on and storing user data in the database system. This non-consensus-based replication protocol typically requires external coordination in the event of node failures, but is faster and more efficient than consensus-based protocols. A state machine can be used to implement changes to the database system configuration based on the configuration data. Using a state machine in this manner ensures that the process of modifying the database system configuration can be processed in parallel with operations performed on user data and does not significantly interrupt or hinder operations performed on the user data. Additionally, consensus-based replication using system configuration data may allow for a faster, but less durable, asynchronous replication protocol to be implemented for processing user data without significant loss of data and transactions in the event of node failures.

[0009] According to a second aspect of the present disclosure, a database management system is provided, which includes: at least one processor; and at least one memory, the memory including computer-executable instructions, which, when executed by the at least one processor, cause the processor to: configure a first group of nodes of the database management system to: determine a transient state of the database management system; generate system configuration data representing a target configuration of the database management system based on the transient state; and provide the system configuration data to a second group of nodes of the database management system, wherein the system configuration data is replicated between the first group of nodes according to a consensus-based replication protocol so as to maintain the system configuration data; and configure the second group of nodes to: receive the system configuration data from the first group of nodes; and implement a state machine, which is arranged to modify the configuration of the second group of nodes according to the system configuration data, wherein the second group of nodes is arranged to replicate user data according to a non-consensus-based replication protocol.

[0010] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which includes computer-executable instructions, which when executed by at least one processor cause the at least one processor to: configure a first group of nodes of a database management system to: determine a transient state of the database management system; generate system configuration data representing a target configuration of the database management system based on the transient state; and provide the system configuration data to a second group of nodes of the database management system, wherein the system configuration data is replicated between the first group of nodes according to a consensus-based replication protocol to maintain the system configuration data; and configure the second group of nodes to: receive the system configuration data from the first group of nodes; and implement a state machine, which is arranged to modify the configuration of the second group of nodes according to the system configuration data, wherein the second group of nodes is arranged to replicate user data according to a non-consensus-based replication protocol. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram illustrating a database system implementing an instance of a database according to an example.

[0012] Figure 2 is a schematic diagram illustrating a database system implementing an instance of a database according to an example, wherein the database system is a distributed database system.

[0013] Figure 3 is a flowchart illustrating a method according to an example.

[0014] Figure 4 is a schematic diagram illustrating a database system including a first group of nodes and a second group of nodes according to an example.

[0015] Figure 5is a schematic diagram illustrating a consensus-based replication protocol according to an example.

[0016] Figure 6 is a schematic diagram illustrating synchronous and asynchronous coordinator nodes according to an example.

[0017] Figure 7 is a schematic diagram illustrating a database system including a first group of nodes and a second group of nodes, wherein user data is divided into a plurality of partitions.

[0018] Figure 8 is a schematic diagram illustrating the functionality of a state machine according to an example.

[0019] Figure 9 is a schematic diagram illustrating predetermined actions taken by the second set of nodes in response to changes in local configuration data, according to an example.

[0020] Figure 10 is a schematic diagram illustrating a non-transitory computer-readable storage medium according to an example. DETAILED DESCRIPTION

[0021] The following description of the systems and methods will be readily apparent from the following description, with reference to the accompanying drawings. For purposes of explanation, this description includes numerous specific details of certain examples. Reference in this specification to "example" or similar language means that a particular feature, structure, or characteristic described in connection with that example is included in at least one example, but not necessarily in other examples. It should also be noted that certain examples are schematically described, with certain features omitted and / or necessarily simplified, to facilitate explanation and understanding of the concepts underlying the examples.

[0022] A database can generally be thought of as an organized collection of data stored electronically in a computer system. A computer system that stores and manages a database and the data stored in it is often referred to as a database management system or database system. In addition to storing databases, database systems can also be configured to perform certain management functions, such as replicating the database to provide resilience against crashes and executing operations to record new data in the database and / or modify data already stored in the database. Some database systems also provide interfaces that enable users to implement workloads configured to perform operations on the data stored in the database.

[0023] Certain properties are often desirable in database systems, such as durability and high availability. Durability refers to a property of a database system that ensures that transactions and user data are permanently stored and not accidentally deleted or erased, even in the event of a typical hardware failure or database crash. High availability refers to a property of a database system in which the database's uptime or availability is higher than normal. Hardware failures can occur in a database system and typically result in a period of downtime in which the database or at least some of the user data stored therein is inaccessible. During this downtime, the database system can be reconfigured, backups restored, transactions replayed, etc. Extended downtime of a database system can cause serious problems for users who rely on the database, especially in time-sensitive and / or high-demand applications. The durability and high availability of a database system can depend on how it is configured to handle, back up, and restore user data.

[0024] High availability can be achieved by implementing a distributed database system architecture, in which the database system is spread across multiple nodes. Nodes can be physical or virtual machines, which together form a single logical database system. Each node in a distributed database system stores a portion of the data, and the nodes work together to manage and process the data. By distributing data across multiple nodes, if one node fails, the others can continue to operate, thus achieving a high level of fault tolerance. Storing different portions of data on different nodes is often referred to as partitioning or implementing a partitioned database. Storing copies of the same data on multiple nodes is often referred to as implementing a replicated database. A combination of these database architectures can be used to create a hybrid database system, in which data is partitioned and copies of the partitions are stored on multiple nodes. Storing replicas of data on different nodes ensures that if a node fails, the database system can move any active workloads or queries to one or more other nodes that have copies of the data stored on the failed node. The ability to quickly switch to a different node without significant downtime provides high availability for the database system.

[0025] Configuring a distributed database system that provides high availability is significantly more complex than configuring a single-node database due to the need to track the location and dependencies of data, ensure backups are up-to-date, and, in some cases, distribute the workload to the nodes where the relevant data is stored. There are many technical challenges that need to be overcome to make a database system sufficiently fault-tolerant and durable so that it can provide high availability. Specifically, obstacles to ensuring that replicas of data stored in the database are up-to-date can include network partitions in the database system that prevent different parts of the database system from seeing the latest changes to the data. Other obstacles include adding or removing nodes from the database system, and nodes failing and subsequently rejoining the database system. When a node is added, fails, and / or rejoins, it can be complex to determine which copies of data the node should or should not have, and in the case of a rejoin, to determine how up-to-date the node was before the failure and what data the node is missing.

[0026] As briefly described above, a replicated database stores multiple copies of data on different nodes so that if one node fails, a different node can take its place. Different replication processes are possible, and certain properties of the database will depend on the specific process used. For example, some replication programs execute asynchronously with transactions. This means that a first node can execute and commit a transaction before the associated data has been replicated to a second, backup node. This allows users to quickly instruct and execute operations, but can make recovering data after a failover of the first node more difficult, as the second node may not have the most up-to-date copies of all user data.

[0027] Other replication procedures, such as synchronous replication or consensus-based replication, can be more powerful, but typically at the expense of slowing down the operations that users instruct the database system to perform. Consensus-based replication involves methods designed to ensure that copies of a shared resource (in this case, user data) are consistently maintained and transactions are not lost, provided that at least a majority of the system resources remain online. This is typically performed using a group of nodes (or replicas of the database) that work together to reach a consensus on the current state of the shared resource. The process of reaching consensus on the current state of the shared resource among the group of nodes can use a variety of suitable algorithms, such as Paxos or Raft.

[0028] Because consensus-based protocols require agreement among nodes replicating data, the process of recording and executing new transactions can be slowed down. This is because, in at least some cases, consensus must be reached on the current state of a resource before changes to that resource (e.g., as a result of an operation or transaction) can occur. Input and output (I / O) and transaction processing can be delayed when more than one node needs to receive and process a transaction for it to complete. Synchronous replication programs can also be subject to similar performance limitations, as transactions cannot be committed until they have been replicated to replicas.

[0029] Compared to synchronous non-consensus-based replication protocols, consensus-based replication protocols typically require significant computational resources to achieve similar levels of durability, making consensus-based replication more expensive. For example, to achieve resilience to one node failure, a synchronous non-consensus-based replication protocol requires two replicas, while a consensus-based replication protocol requires three replicas, representing a 50% increase in resource requirements. In general, to provide resilience to N node failures, a synchronous non-consensus-based replication protocol requires N+1 replicas, while a consensus-based replication protocol may require 2N+1 replicas. While synchronous replication protocols typically require external coordination to manage failover events, consensus-based replication protocols may include procedures for managing failover without external coordination.

[0030] Certain examples described herein provide systems and methods for operating a database management system, designed to manage the configuration of the database system to provide high availability while also maintaining fast and efficient procedures for performing operations on user data. This is achieved by ensuring that data in the database system is effectively replicated and that changes to the database configuration can be implemented quickly, ensuring that if any given node fails, the database system can quickly fail over to one or more backup nodes, thereby minimizing downtime. In some parts of the database system, it may be advantageous to use a combination of persistence and high availability, but with computationally expensive and slow procedures, while in other parts of the database system, it may be advantageous to use a less available, but computationally inexpensive, and fast procedure.

[0031] Figure 1 1 shows a database management system 100 that includes a set of one or more processors 102, a storage device 104, and one or more communication modules 106. The processors 102, the storage device 104, and the communication modules 106 are connected via an interface 108 (e.g., a bus). The processor 102 includes any suitable combination of processing circuits that can be configured to perform the operations described herein. Figures 2 to 9For example, processor 102 may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a fixed function processing circuit, any other suitable processing circuit, and / or any combination thereof.

[0032] Storage 104 may include volatile storage or a combination of memory and non-volatile storage, which may be referred to as disk storage. Volatile storage may include a combination of random access memory (RAM), static random access memory (SRAM), and / or various types of dynamic random access memory (DRAM). While volatile storage has been described with respect to storage 104 in some instances, some or all of the volatile storage may be directly attached to or integrated with processor 102.

[0033] Although sometimes referred to as "disk" or "disk storage," non-volatile storage may include any suitable combination of non-volatile storage types. For example, non-volatile storage may include a hard disk drive, a solid-state drive, flash memory, tape storage, or any combination of these storage types. In some cases, non-volatile storage may include tiered storage in which different storage types are used to store data depending on the age and / or frequency of access of the data. For example, a solid-state drive may be used to store data that is frequently accessed or is likely to be frequently accessed. In contrast, a hard disk drive or tape storage, which has a higher read / write latency than a solid-state drive or flash memory, may be used to store data that is older or less likely to be accessed frequently.

[0034] Storage device 104 is used to store computer-executable instructions 110 and a database 112. When executed by processor 102, computer-executable instructions 110 may implement one or more programs. These programs may include general database management programs for maintaining data in database 112, performing backup and replication processes, providing an interface to database 112, etc. Computer-executable instructions 110 may also include instructions for implementing user-defined workloads. Database 112 included in storage device 104 may include a collection of structured data in a combination of row-store and column-store formats, as well as a collection of unstructured data such as binary large objects.

[0035] The communication module 106 may include any suitable combination of a network-enabled communication module and / or a user interface. For example, the communication module 106 may include a network interface to enable the database system 100 to communicate with one or more additional computer systems 114 via a network 116, such as a local area network, a wide area network (such as the Internet), or the like. Alternatively or in addition, the communication module 106 may include a user interface, such as an input and output device, such as a screen, a keyboard, a touch screen, or the like. The database system 100 may operate and / or be configured based on instructions received from the one or more additional computer systems 114 via the network 116.

[0036] Although the database system 100 Figure 1 Although shown as a single device in FIG. 1 , it should be understood that database system 100 may comprise a collection of independent computing systems. For example, database system 100 may be implemented on one or more collocated servers or clustered computers. Alternatively, database system 100 may be a distributed database system, implemented as multiple computers or servers, remote from one another and communicatively coupled via a network.

[0037] Briefly go to Figure 2 , which shows an example of an arrangement in which the database system 100 is a distributed database system. In this example, the database system 100 includes two server computers 202A and 202B connected via a network 116. The two servers 202A and 202B each include a corresponding set of computing resources, which are divided into different nodes 204A to 204F. Each node is allocated its own set of one or more processors, storage devices, and communication modules for communicating with other nodes in the database system 100. Such as Figure 2 The distributed database system shown may be capable of adding or "spinning up" nodes by adding new resources (such as new servers or allocating new resources in existing servers) or by reconfiguring existing resources.

[0038] Return briefly to Figure 1 The computer executable instructions 110 include a computer executable instruction set 118 that, when executed by the processor 102, causes the database system 100 to perform Figure 3 The computer-implemented method 300 for operating the database management system 100 shown in FIG. 3 is described below with respect to Figures 3 to 9 Further description.

[0039] refer to Figure 3 and Figure 4, the method 300 includes configuring a first set of nodes 402 of the database management system 100 to determine 304 a transient state of the database management system 100, generating 306 system configuration data 404, and providing 308 the system configuration data 404 to a second set of nodes 406 of the database management system 100. Figure 4 In the example shown, there are four nodes 408A to 408D in the first group of nodes 402, including a master node 408A and three coordinator nodes 408B to 408D. It should be understood that this number is given as an example only, and the first group of nodes 402 can have more or less than four nodes. In some examples not shown, the first group of nodes 402 includes at least three nodes 408A to 408D, including a master node 408A and two coordinator nodes 408B and 408C. The first group of nodes 402 can be responsible for managing the configuration and operation of the second group of nodes 406, which can be configured to process user data. The second group of nodes 406 is also referred to as operating nodes 406 in this article.

[0040] Figure 4 The operation nodes 406 (comprising three nodes 410A to 410C) shown in FIG are configured to store and process user data 412. For example, the operation nodes 406 may be configured to receive transaction requests and perform operations on the user data 412 in accordance with these requests. The operations performed in accordance with these transaction requests may include reading, writing, or otherwise modifying the user data 412. In some cases, running a workload or program on the database system 100 causes the operation nodes 406 to perform multiple operations or transactions on the user data 412. The user data 412 may include data that a user has uploaded to the database system 100 or otherwise stored on the database system 100 for organization and / or for use in executing transactions. In addition, the user data 412 may include data generated as a result of operations or transactions performed by the operation nodes 406.

[0041] Operation nodes 406 can be arranged according to a hierarchy or data dependency, where one node 410A controls a primary replica of user data 412, while a second node 410B controls a secondary replica of user data 412. In this case, second node 410B is configured to replicate user data 412 from the primary replica, including any modifications or changes made to the primary replica of user data 412 stored on first node 410A. Designating primary and secondary replicas of user data 412 ensures data consistency across database system 100, thereby preventing transaction or data conflicts. Typically, operations (also known as transactions) are performed on the primary replica of user data 412 and replicated to secondary replicas.

[0042] System configuration data 404 is replicated 310 among first set of nodes 402 according to a consensus-based replication protocol to maintain system configuration data 404. As described above, a consensus-based replication protocol is a type of replication protocol that relies on consensus among a set of nodes to ensure data consistency and can improve the durability, consistency, and high availability of system configuration data 404.

[0043] For example, a consensus-based replication protocol can ensure that a majority of nodes in the first group of nodes 402 have the latest copy of the system configuration data 404, so that when a failover occurs to any node in the first group of nodes 408A to 408D (including the master node 408A), the latest copy of the system configuration data 404 will not be lost.

[0044] If a non-consensus-based replication protocol is used to replicate system configuration data 404 between the first set of nodes 402, different parts of the database system 100 may store different versions of the system configuration data 404. In the event of a failure, it will be difficult to determine which version of the system configuration data 404 is correct, and therefore difficult to determine how to recover any failed nodes 410A to 410C. Figure 5 A specific example of a consensus-based replication protocol is further described.

[0045] The system configuration data 404 represents the target configuration of the database management system 100 based on the instantaneous state. For example, the system configuration data 404 may specify the storage location of the primary and secondary replicas of the user data 412 in the database system 100, the roles of certain operational nodes 410A to 410C, and a list of other related databases. This may include specifying which nodes 410A to 410C the primary and secondary replicas are to be stored on, with the goal of ensuring that the user data is consistent, persistently stored, and highly available in the event of a failover of any one or more of the nodes 410A to 410C. Depending on the instantaneous state, the target locations of the primary and secondary replicas of the user data 412, as specified in the system configuration data 404, may differ from the current locations of the primary and secondary replicas of the user data.

[0046] Determining the instantaneous state of database system 100 can include monitoring the state of one or more nodes in database system 100 and / or the state of data stored thereon. For example, first set of nodes 402 can monitor operational nodes 406 to determine which of nodes 401A-410C are communicating with other nodes (or are "online"), whether any of nodes 410A-410C have lost communication with other nodes (or are "offline"), whether any of nodes 410A-410C have experienced data errors during transactions, and / or whether secondary replicas of user data 412 are up-to-date and / or are replicating quickly enough to ensure high availability.

[0047] Monitoring the status of the operational nodes 406 may include periodically or continuously communicating with the operational nodes 406 to request status updates. The operational nodes 406 may each store one or more variables indicating their respective status. In this case, monitoring the operational nodes 406 may include receiving an indication of the one or more variables. The first group of nodes 402 may receive status information from one or more operational nodes 406, the status information including any one or more of the following: which node in the second group of nodes includes a primary replica of the user data; which node in the second group of nodes 406 includes a secondary replica of the user data; a synchronization status of a node storing a secondary replica of the user data 412; and which node in the second group of nodes 406 is currently communicating with at least one other node in the second group of nodes 406.

[0048] Determining this state information allows first set of nodes 402 to identify where user data 412 is stored, which node in second set of nodes 406 should replicate user data 412 from other nodes, and whether any one of the nodes in second set of nodes 406 has crashed or is unable to communicate with another node in second set of nodes 406. When a node in second set of nodes 406 is no longer communicating with any other node in second set of nodes 406, it may be necessary to recover it so that it can continue replicating to and / or from other nodes in second set of nodes 406.

[0049] The operational nodes 406 may additionally or alternatively be configured to transmit state information to the first set of nodes 402 triggered by a state change. Configuring the operational nodes 406 to proactively transmit state information enables the system 100 to react more quickly to problems, such as replication failures or loss of synchronization. When a secondary replica is unable to confirm a transaction in synchronous replication, the primary replica may not expect to commit the transaction until the secondary replica is marked as asynchronous by the first set of nodes 402. Otherwise, if a failover is triggered before the first set of nodes 402 is aware of the data loss on the secondary replica, there may be data loss. Therefore, quickly identifying state changes and reacting to state changes can mitigate the risk of data loss in these situations. The responsibility for monitoring the database system 100 to determine the instantaneous state can be borne by the master node 408A. Configuring only one of the first set of nodes 402 to determine the instantaneous state of the database can eliminate the possibility of inconsistencies between the first set of nodes 402 regarding the state of the database system 100, which might otherwise occur due to, for example, communication delays or errors.

[0050] As the transient state of database system 100 evolves, the state may be communicated to different nodes in first set of nodes 402 at different rates, and thus coordination between first nodes 402 in this regard may not be necessary. When master node 408A is responsible for determining the transient state of database system 100, master node 408A may have a communication state with second set of nodes 406. Master node 408A may determine or detect changes in the communication state with second set of nodes 406 to determine the transient state of database system 100. For example, the communication state may specify which nodes in second set of nodes 410A-410C are in communication with master node 408A. Changes in the communication state may include a loss of communication with one or more nodes in second set of nodes 406, a change in communication latency with one or more nodes in second set of nodes 402, or some other change that affects communication between master node 408A and operational nodes 410A-410C. Master node 408A may implement a heartbeat protocol with second set of nodes 406. For example, according to the protocol, if one node in the second set of nodes 406 (eg, node 410B) fails to send a signal to the master node 408A within a predetermined interval, the master node 408A may determine that the node 410B has lost communication with the master node 408A.

[0051] The second set of nodes 406 (also referred to as operational nodes) are configured 312 to receive system configuration data 404 from the first set of nodes 402 and implement 316 one or more state machines 414A through 414C that are arranged to modify the configuration of the second set of nodes 406 based on the system configuration data 404. The first set of nodes 402 may provide the system configuration data 404 to the second set of nodes 406 by transmitting the system configuration data 404 via the network 112 or bus 108, depending on the architecture of the system 100. In some cases, the system configuration 404 may be provided to the second set of nodes 406 based on a replication protocol, but typically this will be a non-consensus-based replication protocol.

[0052] exist Figure 4 In the illustrated example, each of the second set of nodes 410A-410C implements a corresponding state machine 414A-414C, but it should be understood that a state machine can be shared between two or more of the second set of nodes 406. By implementing state machines 414A-414C in this manner, the second set of nodes 406 are able to take actions to place the database system 100 in a configuration according to the system configuration data 404. By instructing each node in the second set of nodes 406 to take local actions to modify the configuration of the database system 100, the system 100 becomes more responsive to failures and transient state changes. When a local failure occurs in the second set of nodes 406, if the failure is recoverable, the relevant nodes in the second set of nodes 406 can return to the global configuration state according to the system configuration data 404 without involving other nodes. This, in turn, reduces the monitoring burden and the latency in reacting to node failures.

[0053] One such example involves a first node 410A having a primary replica of user data 412 and a second node 410B having a secondary replica of the user data 412. If the system configuration data 404 specifies that the location of the primary replica is the second node 410B and the location of the secondary replica is the first node 410A, then, contrary to the current state of the database system 100, the state machines 414A and 414B can modify the configuration of the first node 410A and the second node 410B to change which of the nodes has the primary replica of the user data 412 and which node has the secondary replica.

[0054] Modifying the configuration of the database system 100 in this case may involve providing the second node 410B with the right to execute transactions on the user data 412 stored thereon and causing the first node 410A to replicate the user data 412 from the second node 410B. Other operations may be performed when modifying the location of the primary replica of the user data 412. The changes required to the configuration of the second set of nodes 406 may depend on the structure and arrangement of the second set of nodes 406 and the manner in which the database system 100 is configured to operate.

[0055] In this example, the first set of nodes 402 is responsible for centrally determining the desired configuration of database system 100, aiming to ensure high availability of user data 412. As new nodes are added to operational nodes 406, nodes fail or go offline, and other state changes occur that could disrupt replication dependencies between operational nodes 406, the first set of nodes 402 can send configuration data to operational nodes 406 to address these changes. By enabling the second set of nodes 406 to autonomously arrange themselves according to the desired configuration based on system configuration data 404, the configuration and reconfiguration of database system 100 is democratized, making system 100 more resilient and more adaptable to the addition and removal of new nodes. This is because the first set of nodes 402 is not responsible for actively coordinating replication and / or other communications between the second set of nodes 406. Distributing the workload involved in managing new nodes reduces the likelihood that any one node will be overwhelmed by processing system configuration changes and / or handling I / O requests from multiple other nodes in the system to coordinate the configuration.

[0056] As described above, consensus-based replication protocols may require greater computing resources (e.g., more nodes) to achieve the same level of durability as certain synchronous replication protocols. While consensus-based replication protocols do provide higher data availability, this increased expense may make it undesirable to perform operations on user data 412 using a consensus-based replication protocol. Therefore, user data 412 may be replicated 318 by a second set of nodes 406 according to a non-consensus-based replication protocol. Using a more flexible, non-consensus-based replication program to replicate user data 412 may improve the performance of the operating nodes 406 when operating on the user data 412. This also makes the second set of nodes 406 more suitable for handling highly transactional workloads that would be cumbersome to execute in an environment that relies on a consensus-based replication protocol to process user data 412.

[0057] In some database system 100 configurations, the number of second set of nodes 406 may be much greater than the number of first set of nodes 402. Therefore, it is desirable to provide operational nodes 406 that can process transaction requests and replicate in an agile manner because the likelihood and frequency of node failover or horizontal scaling of the second set of nodes 406 are much higher than that of the first set of nodes 402.

[0058] As mentioned above about Figure 4As shown, the first set of nodes 402 includes a master node 408A and two or more coordinator nodes 408B through 408D, which are communicatively coupled to the master node 408A and configured to replicate system configuration data 404 according to a consensus-based replication protocol. The two or more coordinator nodes 408A through 408D may also be referred to as voting members or voting member nodes, indicating that they are entitled to vote during the consensus-based replication protocol, where their votes represent whether they agree with changes made to the system configuration data 404. The consensus-based replication protocol may specify that the generation of the system configuration data 404 is initiated by the master node 408A, and that the generation of the system configuration data 404 is contingent upon a majority of the nodes in the first set of nodes 402 reaching a consensus with the master node 408A regarding the system configuration data 404. In this manner, changes to the system configuration data 404 can be made when they can be durably stored in the database system 100, while changes to the system configuration data 404 can be prevented when the durability of the changes cannot be guaranteed.

[0059] Go to Figure 5 , shows an example of a consensus-based replication protocol. In this example, generating the system configuration 404 includes the master node 408A generating a write request 504 to modify the system configuration data 404 based on the instantaneous state of the database system 100. Figure 5 As can be seen in FIG, system configuration data 404 initially includes a set of entries or data entries 502A through 502C. These entries may specify certain properties related to the configuration of database system 100, such as where portions of user data 412 are stored and which nodes 410A through 410C store primary and secondary copies of user data 412. These entries 502A through 502C may have been determined at a previous point in time, for example, during a previous operation that generated system configuration data 404. A write request 504 for modifying system configuration data 404 may include an indication that a new entry 502D is to be added to system configuration data 404. Alternatively or in addition, write request 504 may indicate modifications to previous entries 502A through 502C.

[0060] Coordinator nodes 408B through 408D process request 504 to replicate the write request from master node 408A to coordinator nodes 408B through 408D. The presence or absence of new entry 502D indicates that two of coordinator nodes 408B and 408C have successfully processed the write request, while the third coordinator node 408D has not yet successfully replicated the write request 504. This may be due to a processing delay or some other failure, such as a data error, a software problem, or a network failure when communicating with the third coordinator node 408D.

[0061] Depending on the result of processing the write request 504 using the coordinator nodes 408B to 408D, the step of generating the system configuration data 404 can be performed in one of two ways. If at least half of the coordinator nodes 408B to 408D successfully replicate the write request 504 from the master node 408A to at least half of the coordinator nodes 408A to 408D, the system configuration data 404 is modified according to the write request 504. If at least half of the coordinator nodes 408B to 408D do not successfully replicate the write request 504 from the master node to at least half of the coordinator nodes 408A to 408D, the system configuration data 404 is not modified according to the write request 504. In other words, the write request 504 is rejected. In this way, at least half or all of the first group of nodes 402 (including the master node 408A) need to have a copy of the write or modification to be performed in order to complete these changes. Halting writes to the system configuration data 404 ensures that a single version of the latest copy of the system configuration data 404 is known to at least half of the remaining coordinator nodes 408B through 408D in the event of a failover (eg, of the master node 408A).

[0062] As described above, in some cases, any coordinator node that has replicated / non-replicated the write request 504 may experience delays when communicating with the master node 408A or when processing the write request 504 on the node. In such cases, the node may then replicate the write request 504. For example, the write request 504 may be accepted by a majority of the first set of nodes 402 and subsequently accepted by any remaining nodes, such as node 408D in this example.

[0063] In some consensus-based replication protocols, for example, consensus-based replication protocols based on or similar to the Raft protocol, Figure 5The coordinator nodes 408B to 408D shown, or voting member nodes, may be referred to as in-sync coordinator nodes, meaning that, when the request 504 for system configuration data 404 is generated, these nodes include the latest copy of the current system configuration data 404 as stored on the master node 408A. For example, the in-sync coordinator nodes 408B to 408D include data entries 502A to 502C representing the database system configuration. In some cases, the in-sync coordinator nodes 408B to 408D may additionally have an active or confirmed communication channel with the master node 408A to receive the write request 504. The first group of nodes 402 may additionally include one or more asynchronous coordinator nodes for storing a replica of the system configuration data 404 that is not synchronized with the replica of the system configuration data 404 stored on the in-sync coordinator nodes 408B to 408D. In some examples, these asynchronous coordinator nodes are nodes that, in addition to not having an in-sync copy of the system configuration data 404, have also experienced a communication failure with the first group of nodes 402 and are therefore currently unable to participate in voting.

[0064] Figure 6 Shown Figure 5 , and node 408D, which failed to successfully replicate write request 504, has become asynchronous coordinator node 602. As can be seen here, synchronous coordinator nodes 408B through 408C each store a contemporary copy of system configuration data 404 that matches the system configuration data 404 stored on master node 408A. Asynchronous coordinator node 602 does not have a contemporary copy of system configuration data 404 because, in this example, it lacks the last data entry 502D. For a variety of reasons, coordinator nodes 408B through 408D may become out of sync with the other coordinator nodes. For example, due to data corruption, loss of network connectivity, or temporary resource reallocation, coordinator node 408D may be unable to successfully process and record one or more write requests 504 generated by master node 408A. In some cases, the out-of-sync coordinator node 408D can quickly resynchronize with the remaining nodes 408A through 408C. However, in some cases, they may remain asynchronous for an extended period of time. When the synchronization coordinator node 408D loses synchronization with the remaining coordinator nodes 408B and 408C (e.g., Figure 5As shown, where the last node 408D is unable to process the write request 504 and is unable to regain synchronization), it may no longer be given a vote in the subsequent consensus-based replication process. Removing the voting rights of the coordinator node 408D can be initiated by the master node 408A and may involve modifying the number of nodes 402 required to reach a majority consensus. The procedure can be performed when the coordinator node has been asynchronous for a predetermined period of time and / or when the coordinator node is no longer communicating with some or all of the remaining coordinator nodes 408A to 408C. This allows the consensus-based replication process to proceed without noticeable delay when one or more coordinator nodes 408B to 408D lose the ability to communicate or vote with the remaining coordinator nodes 408B to 408D.

[0065] The system configuration data 404 can be sent to the synchronous coordinator nodes 408B and 408C and the asynchronous coordinator node 408D to enable the asynchronous coordinator node 408D to catch up with the synchronous coordinator nodes 408B and 408C. In an example in which voting rights are removed from the asynchronous coordinator node, the first group of nodes 402 can be further configured to monitor the synchronization status of one or more asynchronous coordinator nodes 602. If a given asynchronous coordinator node 602 obtains synchronization with the synchronous coordinator nodes 408B to 408C as determined in the monitoring, the asynchronous coordinator node 602 can become the synchronous coordinator node and be provided with a vote in a subsequent consensus-based replication procedure. One or more additional replication procedures can be performed between the first group of nodes 402 to resynchronize the asynchronous nodes 602. This may involve performing a snapshot recovery or other non-consensus-based replication protocol. The replication procedure can be based on data from one of the synchronous coordinator nodes 408B to 408C and / or the master node 408A.

[0066] In this way, if the coordinator node (such as Figure 5 408D) loses synchronization, it can be resynchronized with the coordinator node group 408B to 408C. The consensus-based replication protocol relies on a minimum number of nodes that can vote on the consensus on the write of new system configuration data 404. Therefore, being able to maintain voting member nodes allows a minimum number of voting members to be maintained among the coordinator nodes 408B to 408D. In some cases, such as due to a network failure, it may be the case that there are not enough voting members in communication with the master node 408A available to vote meaningfully for the purpose of reaching consensus. In this case, the master node 408A can continue to record transactions or changes to the system configuration 404, but they will not be completed until a sufficient number of voting members are back online.

[0067] Other consensus-based replication protocols (such as those based on or similar to Paxos) may proceed in different ways. Specifically, in Paxos, if each write request 504 is acknowledged by a majority of nodes, then each coordinator node may not need to have a synchronized copy of all system configuration data 404.

[0068] The election of the master node can be performed according to an election procedure in which the master node is selected by the first group of nodes 402. The master node can initially be a coordinator node and can be elected as the master node by consensus among the first group of nodes 402. If the master node fails, the remaining first group of nodes 402 can elect a new master node. If a failure condition associated with the master node is detected, two or more coordinator nodes are configured to elect a coordinator node as the new master node.

[0069] There are a variety of different types of election procedures that can be performed by the first group of nodes 402 to elect a new master node. In some cases, the election may include identifying which coordinator nodes are synchronization coordinator nodes, and then selecting nodes from the synchronization coordinator nodes. The selection of nodes from the synchronization coordinator nodes can be based on the available resources of each of the synchronization nodes. As described above, the master node can perform functions that exceed those of the other nodes in the first group of nodes 402, such as determining the instantaneous state of the database, initiating a write request 504 for generating system configuration data 404, etc. Therefore, it is desirable to select the node with the greatest available resources. In some examples, the election may be performed based on a simple majority or an absolute majority. In some cases, the voting member nodes may be limited to selecting one coordinator node from a subset of all available coordinator nodes.

[0070] Apply it to Figure 4 In the example shown, the master node 408A is the coordinator node elected by the majority of the coordinator nodes 408A to 408D. If a fault condition associated with the master node 408A is detected, the remaining coordinator nodes 408B to 408D will elect one of the remaining coordinator nodes 408B as the new master node. The coordinator node 408B selected as the new master node is a synchronous coordinator node, which means that it has a contemporary copy of the system configuration 404 and can therefore continue the operation of the previous master node 408A without interruption. The fault condition may include a communication failure between the master node 408A and the remaining coordinator nodes 408B to 408D and / or the second group of nodes 402. Alternatively or in addition, the fault condition may include a fault or data error in the master node 408A.

[0071] In the above about Figures 4 to 6In the depicted example, user data 412 is shown as a single data block stored on nodes 410A to 410C. Alternatively, user data 412 may be split into two or more partitions to increase storage and / or computational resiliency, that is, to increase the ability to scale database system 100, thereby allowing additional resources to be used to store data and / or execute queries or other transactions on the data stored therein. Figure 7 , shows an example of dividing user data 412 into three partitions, each partition including a portion of the entire user data 412. Primary replicas 702A to 702C of these partitions are each stored on a different node 410A to 410C in the second set of nodes 406. Secondary replicas 704A to 704C of the partitions are also distributed across the second set of nodes 410A to 410C. The secondary replicas 704A to 704C can be stored on a different node in the second set of nodes 406 than their corresponding primary replicas 702A to 702C. Partitioning the data to distribute user data 412 across the second set of nodes 402 provides better availability in the event of a failover of any of the second set of nodes 410A to 410C.

[0072] When the database system 100 is configured as Figure 7 When user data 412 is partitioned as shown, method 300 is modified somewhat. Specifically, determining the instantaneous state of database system 100 can include determining the location of the primary replica and the secondary replica for each of the partitions in database system 100. For example, when user data is split into two or more partitions, each having a primary replica and a secondary replica, an indication of which node in second set of nodes 406 includes the primary replica and / or the secondary replica of user data 412 can be provided for each partition. Additionally or alternatively, the synchronization status of the secondary replica of user data 412 can be provided for each partition. In this way, granular details of the precise state of each of the primary replica and the secondary replica of a partition of user data 412 can be determined and used when generating system configuration data 404.

[0073] Figure 84. Operation of the state machine 414A according to an example is shown. An operational node 410A may store local configuration data 802 representing a configuration of a second set of nodes. The local configuration data may represent an identifier of the corresponding node, a role of the node 410A (which may include any replication or management functions for which the node 410A is responsible), a synchronization state of any replicas stored on the node 410A, an indication of a log sequence number (or "LSN") (indicating a recently executed transaction), an indication of a previous failure, and the like. In some examples, each of the nodes 410A may include local configuration data 802 representing information related to that particular node 410A. In other examples, the local configuration data 802 stored on the node 410A may represent a configuration of a set of one or more nodes in the second set of nodes 406.

[0074] State machine 414A can be configured to process system configuration data 404 received from first set of nodes 402 to identify one or more differences between local configuration data 802 and system configuration data 404. For example, system configuration data 404 can include a plurality of data entries 804A through 804N representing a target configuration for database system 100. Local configuration data 802 can include a plurality of entries 806A through 806N representing a local configuration associated with node 410A. State machine 414A can process system configuration data 404 and local configuration data 802 and identify that a characteristic of the local configuration, as specified by a particular data entry 806C, conflicts with or differs from a corresponding entry 804C in system configuration data 404. For example, data entry 806C in local configuration data 802 can specify that partition 702A is to be replicated to first node 410B, while data entry 804C in system configuration data 404 can specify that partition 702A is to be replicated to a different second node 410C.

[0075] State machine 414A may modify local configuration data 802 based on one or more differences. In this case, data entry 806C may be modified to generate a new data entry 808 in local configuration data 802 that does not conflict with the characteristics represented by corresponding entry 804C in system configuration data 404. Modifying local configuration data 802, and in particular first data entry 804C, may include overwriting one or more data values. Alternatively, one or more portions 806C of local configuration data 802 may be deleted, and new data portions 808 may be generated based on system configuration data 404 (and in particular data entry 804C). Local configuration data 802 may be used by second group of nodes 406 when performing operations, replication, and other functions on user data 412. For example, certain management functions performed by second group of nodes 406 may reference local configuration data 802 when determining how to proceed.

[0076] Modifying the local configuration data 802 may cause the second set of nodes 406 to perform a predetermined action. Figure 9 The following example is shown: a partition 704A stored on node 410B has gone offline (indicated by a communication failure 902 with partition 704A), meaning it is not currently replicating primary replica 702A and / or may be in an asynchronous state, meaning it is not keeping up to date with the latest changes to primary replica 702A. A first set of nodes 402 (and preferably master node 408A) detects this change in the state of secondary replica 704A and provides system configuration data 404 to a second set of nodes 406 to enable nodes 406 to reconfigure themselves to address this failure 902. State machines 414A to 414C may modify local configuration data 802 based on system configuration data 404 to specify one or more changes to the configuration. This may cause a first node 410A in the second set of nodes 406 to replicate 904 user data 702A to a second node 410C in the second set of nodes 406. In this case, this means that a secondary replica 704A of the first partition 702A is generated for the new node 410C when the previously designated secondary replica is no longer replicating the primary replica.

[0077] In some cases, the primary replica 702A may have an error, or the node 410A storing the primary replica 702A may crash. In this case, the predetermined action may include selecting a given node 410C to store the primary replica 702A of the user data, such as by converting a secondary replica 704A of the same portion of the user data to the primary replica 702A.

[0078] In addition to failovers or synchronization errors, other changes in state may cause user data 412 to be moved or replicated between the second set of nodes 406. For example, when a new node is to be added to the second set of nodes 406, system configuration data 404 is generated by the first set of nodes 402, which specifies the role of the new node, the data to be stored, and whether it is responsible for the primary replica, secondary replica, or a combination of replica types of the user data 412. The first set of nodes 402 distributes the system configuration data 404 to the second set of nodes 406 (including the new node) so that they can modify their local configuration data 802 to conform to the configuration specified in the new system configuration data 404. In doing so, the second set of nodes 406 may initiate multiple replication procedures to bring the local configuration into line with the configuration specified by the system configuration data 404. These replication procedures may include replicating the user data 412 to the new node.

[0079] Another example includes when a new user data partition 412 is to be added or generated from a previous partition 702A. This can be initiated by the master node 408A, for example, when the master node 408A determines that the user data 412 needs to be redistributed among the second set of nodes 406 to increase the availability or durability of the user data 412. Alternatively or in addition, the master node 408A can initiate such a redistribution when certain portions of the user data 412 need to be allocated to nodes in the second set of nodes 406 with more computing power to perform operations on the data 412 in order to handle a workload directed by a user. A user of the system 100 can also initiate such a configuration change by providing instructions to the master node 408A (e.g., via the communication module 106). When adding a new partition, the first set of nodes 402 can generate system configuration data 404 that specifies the storage location of the new partition, including its corresponding primary and secondary replicas. The system configuration data 404 is provided to the second set of nodes 406. The second set of nodes 406 can then modify their local configuration based on the system configuration data 404. In this example, this involves generating a new partition, including a primary replica and a secondary replica. One or more nodes in the second set of nodes 406 can then replicate the relevant portion of the data 412 to other nodes in the second set of nodes 406 to populate the primary replica and the secondary replica of the new partition. A similar process can also be performed when a partition is to be deleted or combined with one or more other partitions.

[0080] In some examples, a previous change to the system configuration data 404 (e.g., based on a change in the instantaneous state of the system 100) can trigger the first set of nodes 402 to make further changes to the system configuration data 404. For example, if a first node 410A storing a primary replica 702A of a partition fails, a secondary replica 704A of the partition stored on a second node 401B can be converted to the primary replica. In this example, the second node 410B can store two primary replicas of different partitions 702B and 704A. In this example, the system configuration data 404 can then be modified to specify that at least one of the replicas 702B-704A stored on the node 410B should be converted to a secondary replica and that the primary replica should be stored on a different node 410C. Storing multiple primary replicas of user data 412 on a single node may be undesirable because the risk of downtime in the event of a failure of the node 410B may be higher. Thus, the system configuration data 404 may be used to balance the distribution of the primary and secondary replicas of the user data 412 on the second set of nodes 406 .

[0081] Figure 10A non-transitory computer-readable storage medium 1000 is shown that includes computer-executable instructions 1002A and 1002B that, when executed by one or more processors 1004, cause the one or more processors 1004 to perform the method 300. The first instruction block 1002A, when executed, causes the processor 1004 to configure a first set of nodes 402 of the database management system to determine 1006A a transient state of the database management system 100, generate 1006B system configuration data 404 representing a target configuration of the database management system 100, and provide 1006C the system configuration data 404 to a second set of nodes 406. As described above with reference to Figures 3 to 9 As described, the first set of nodes 402 replicates 1006D the system configuration data 404 among the first set of nodes 402 according to a consensus-based replication protocol.

[0082] The second instruction block 1002B, when executed, causes the processor 1004 to configure the second set of nodes 406 of the database management system 100 to receive the system configuration data 404 and implement a state machine 414A arranged to modify the configuration of the second set of nodes 406 according to the system configuration data 404. The second set of nodes 406 are arranged to replicate 1008C the user data 412 according to a non-consensus-based protocol. Figures 2 to 9 Any additional features or examples described also apply to Figure 10 The implementation shown.

[0083] The above examples should be understood as illustrative. Further modifications and features may also be implemented in accordance with the present disclosure. For example, one or more nodes in the first set of nodes 402 may also be configured to process user data 412, such as by being included in the second set of nodes 406. The consensus-based protocol for replication and / or electing a new master node may implement a timeout feature for voting members, such that voting members must respond to requests within a predetermined time period. Voting members that fail to respond within the consensus timeout period may be removed from voting in the consensus-based replication protocol until they are synchronized with the remaining voting members.

[0084] Additionally, in the examples described herein, the database system 100 implements a single cluster configured to operate on user data 412 that may or may not be partitioned. It should be appreciated that the examples described herein are also applicable to database systems 100 operating multiple node clusters, each responsible for managing and processing corresponding user data 412. Where multiple clusters are provided by the database management system 100, a single first set of nodes 402 can be used to manage the configuration of each of the clusters and, in some cases, to coordinate between the clusters. Alternatively, the database management system 100 can implement a separate first set of nodes 402 for each cluster.

[0085] It should be understood that any feature described with respect to any one example may be used alone or in combination with the other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be employed.

Claims

1. A computer-implemented method for operating a database management system, the method comprising: The first set of nodes of the database management system is configured to: determining a momentary state of the database management system; generating system configuration data representing a target configuration of the database management system based on the instantaneous state; and providing the system configuration data to a second set of nodes of the database management system, wherein the system configuration data is replicated among the first set of nodes according to a consensus-based replication protocol to maintain the system configuration data; and Configure the second set of nodes to: receiving the system configuration data from the first set of nodes; and implementing a state machine arranged to modify the configuration of the second set of nodes in accordance with the system configuration data, Wherein the second set of nodes is arranged to replicate user data according to a non-consensus based replication protocol.

2. The computer-implemented method for operating a database management system of claim 1 , wherein the first set of nodes includes a master node configured to generate the system configuration data and two or more coordinator nodes configured to replicate the system configuration data according to the consensus-based replication protocol, and wherein generating the system configuration data comprises: The master node generates a write request to modify the system configuration data based on the transient state; processing the request using two or more coordinator nodes to replicate the write request from the master node to the two or more coordinator nodes; as well as At least one of the following: if at least half of the coordinator nodes successfully replicate the write request from the master node to the at least half of the coordinator nodes, modifying the system configuration data according to the write request; or If at least half of the coordinator nodes fail to successfully replicate the write request from the master node to the at least half of the coordinator nodes, the write request is rejected.

3. The computer-implemented method for operating a database management system of claim 2 , wherein the coordinator node is a synchronous coordinator node, and the first set of nodes further comprises one or more asynchronous coordinator nodes, each asynchronous coordinator node storing a replica of the system configuration data, and their respective replicas are not synchronized with the replicas of the system configuration data stored in the two or more synchronous coordinator nodes, and wherein the first set of nodes is configured to: monitoring the synchronization status of the one or more asynchronous coordinator nodes; and If the given asynchronous coordinator node is synchronized with the synchronous coordinator node, the given asynchronous coordinator node is promoted to the synchronous coordinator node.

4. A computer-implemented method for operating a database management system according to any one of claims 1 to 3, wherein the first group of nodes includes a master node and two or more coordinator nodes, the two or more coordinator nodes are communicatively coupled to the master node and are configured to replicate the system configuration data according to the consensus-based replication protocol, wherein generation of the system configuration data is initiated by the master node and depends on a majority of the nodes in the first group of nodes reaching a consensus with the master node.

5. The computer-implemented method for operating a database management system of claim 4, wherein if a failure condition associated with the master node is detected, the two or more coordinator nodes are configured to elect a coordinator node as a new master node.

6. A computer-implemented method for operating a database management system according to any one of claims 1 to 5, wherein the user data is stored in the database management system as a primary replica and a secondary replica, wherein operations to be performed on the user data are performed on the primary replica and subsequently replicated to the secondary replica, and wherein determining the instantaneous state of the database management system comprises receiving state information from one or more of the second set of nodes, wherein the state information comprises at least one of: an indication of which node of the second set of nodes comprises the primary replica of the user data; an indication of which node of the second set of nodes includes the secondary replica of the user data; for the nodes in the second set of nodes comprising the secondary replica of the user data, a synchronization status of the secondary replica of the user data; and An indication of which node of the second set of nodes is currently communicating with at least one other node of the second set of nodes.

7. The computer-implemented method for operating a database management system of claim 6, wherein the user data is partitioned into two or more partitions, each partition having a primary replica and a secondary replica, and wherein: providing, for each partition, said indication as to which node of said second set of nodes comprises said primary replica of said user data; providing, for each partition, said indication as to which node of said second set of nodes comprises said secondary replica of said user data; and The synchronization status of the secondary replica of the user data is provided for each partition.

8. A computer-implemented method for operating a database management system according to any one of claims 1 to 7, wherein the first set of nodes comprises: a master node, the master node being configured to generate the system configuration data; and two or more coordinator nodes, the two or more coordinator nodes being communicatively coupled to the master node and configured to replicate the system configuration data according to the consensus-based replication protocol, and wherein the master node has a communication state with the second set of nodes, and determining the instantaneous state of the database management system includes determining a change in the communication state with the second set of nodes.

9. The computer-implemented method for operating a database management system of claim 8, wherein the second set of nodes is configured to: storing local configuration data representing the configuration of the second set of nodes; processing the system configuration data using the state machine to identify one or more differences between the local configuration data and the system configuration data; and The local configuration data is modified based on the one or more differences.

10. The computer-implemented method for operating a database management system of claim 9, wherein modifying the local configuration data according to the one or more differences causes the second set of nodes to perform a predetermined action, the predetermined action comprising at least one of: causing a first node in the second set of nodes to copy the user data to a second node in the second set of nodes; or A given node in the second set of nodes is selected to store a primary replica of the user data.

11. A database management system comprising at least one processor and at least one memory, the memory comprising computer executable instructions that, when executed by the at least one processor, cause the processor to: The first set of nodes of the database management system is configured to: determining a momentary state of the database management system; generating system configuration data representing a target configuration of the database management system based on the instantaneous state; and providing the system configuration data to a second set of nodes of the database management system, wherein the system configuration data is replicated among the first set of nodes according to a consensus-based replication protocol to maintain the system configuration data; and Configure the second set of nodes to: receiving the system configuration data from the first set of nodes; and implementing a state machine arranged to modify the configuration of the second set of nodes in accordance with the system configuration data, Wherein the second set of nodes is arranged to replicate user data according to a non-consensus based replication protocol.

12. The database management system of claim 11 , wherein the first set of nodes includes a master node configured to generate the system configuration data and two or more coordinator nodes configured to replicate the system configuration data according to the consensus-based protocol, and wherein generating the system configuration data comprises: The master node generates a write request to modify the system configuration data based on the transient state; processing the request using two or more coordinator nodes to replicate the write request; as well as At least one of the following: If at least half of the coordinator nodes successfully replicate the write request, modifying the system configuration data according to the write request; or If at least half of the coordinator nodes fail to successfully replicate the write request, the write request is rejected.

13. The database management system of claim 12 , wherein the coordinator node is a synchronous coordinator node, and the first set of nodes further comprises one or more asynchronous coordinator nodes, each asynchronous coordinator node storing a replica of the system configuration data, and their respective replicas are not synchronized with the replicas of the system configuration data stored in the two or more synchronous coordinator nodes, and wherein the computer-executable instructions, when executed by the at least one processor, cause the processor to configure the first set of nodes to: monitoring the synchronization status of the one or more asynchronous coordinator nodes; and If the given asynchronous coordinator node is synchronized with the synchronous coordinator node, the given asynchronous coordinator node is promoted to the synchronous coordinator node.

14. A database management system according to any one of claims 11 to 13, wherein the first group of nodes includes a master node and two or more coordinator nodes, the two or more coordinator nodes are communicatively coupled to the master node and are configured to replicate the system configuration data according to the consensus-based replication protocol, wherein generation of the system configuration data is initiated by the master node and depends on a majority of the first group of nodes reaching a consensus with the master node, and wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to: if a failure condition associated with the master node is detected, configure the two or more coordinator nodes to elect a coordinator node as a new master node.

15. The database management system of any one of claims 11 to 14, wherein the user data is stored in the database management system as a primary replica and a secondary replica, wherein operations to be performed on the user data are performed on the primary replica and subsequently replicated to the secondary replica, and wherein determining the instantaneous state of the database management system comprises receiving state information from the one or more of the second set of nodes, wherein the state information comprises at least one of: an indication of which node of the second set of nodes comprises a primary replica of the user data; an indication of which node of the second set of nodes comprises a secondary replica of the user data; for the nodes in the second set of nodes comprising the secondary replica of the user data, a synchronization status of the secondary replica of the user data; and An indication of which node of the second set of nodes is currently communicating with at least one other node of the second set of nodes.

16. The database management system of claim 15, wherein the user data is partitioned into two or more partitions, each partition having a primary replica and a secondary replica, and wherein: providing, for each partition, said indication as to which node of said second set of nodes comprises said primary replica of said user data; providing, for each partition, said indication as to which node of said second set of nodes comprises said secondary replica of said user data; and The synchronization status of the secondary replica of the user data is provided for each partition.

17. The database management system according to any one of claims 11 to 16, wherein the first group of nodes comprises: a master node, the master node being configured to generate the system configuration data; and two or more coordinator nodes, the two or more coordinator nodes being communicatively coupled to the master node and configured to replicate the system configuration data according to the consensus-based replication protocol, and wherein the master node has a communication state with the second set of nodes, and determining the instantaneous state of the database management system includes determining a change in the communication state with the second set of nodes.

18. The database management system of claim 17, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to configure the second set of nodes to: storing local configuration data representing the configuration of the second set of nodes; processing the system configuration data using the state machine to identify one or more differences between the local configuration data and the system configuration data; and The local configuration data is modified based on the one or more differences.

19. The database management system of claim 18, wherein modifying the local configuration data according to the one or more differences causes the second set of nodes to perform a predetermined action, the predetermined action comprising at least one of: causing a first node in the second set of nodes to copy the user data to a second node in the second set of nodes; or A given node in the second set of nodes is selected to store a primary replica of the user data.

20. A non-transitory computer-readable storage medium comprising computer-executable instructions that, when executed by at least one processor, cause the at least one processor to: The first set of nodes of the database management system is configured to: determining a momentary state of the database management system; generating system configuration data representing a target configuration of the database management system based on the instantaneous state; and providing the system configuration data to a second set of nodes of the database management system, wherein the system configuration data is replicated among the first set of nodes according to a consensus-based replication protocol to maintain the system configuration data; and Configure the second set of nodes to: receiving the system configuration data from the first set of nodes; and implementing a state machine arranged to modify the configuration of the second set of nodes in accordance with the system configuration data, Wherein the second set of nodes is arranged to replicate user data according to a non-consensus based replication protocol.

Images