Access consistency in high availability databases

Abstract

Techniques related to maintaining a high-availability (HA) database are disclosed. In some embodiments, a computer system receives a plurality of requests for access to data from a plurality of host computers, the data stored in a database implemented using a plurality of clusters. In some embodiments, the computer system responds to the plurality of requests by accessing the data stored in an active cluster. The computer system can then determine health information for a cluster of the plurality of clusters based on the response, where the health information is generated based on real-time traffic of the database. In some embodiments, the computer system determines whether to switch from accessing the active cluster to accessing a backup cluster based on the health information. In some embodiments, the computer system stores a transition decision generated based on the determination in each cluster of the database.

Description

Technical Field

[0001] This disclosure relates generally to database systems, and more specifically to techniques for maintaining multiple database clusters (e.g., for high availability). Background Technology

[0002] Databases comprising multiple clusters provide high availability for the computers accessing these databases. For example, two clusters may contain exact copies of the same data. One of these clusters may fail due to health issues, causing access to that cluster to fail. As a result, these clusters may no longer be synchronized (i.e., contain the same data). This phenomenon can be referred to as a "split-brain" situation. This occurs when two different datasets initially synchronized in a database lose their ability to synchronize because one of the datasets becomes inaccessible. Because one dataset is inaccessible, various Data Processing Language (DML) operations sent to corresponding objects in the two datasets of the database (e.g., the primary cluster and the backup cluster) will only be processed on the healthy dataset. Attached Figure Description

[0003] Figure 1 This is a block diagram illustrating an example database system configured to manage databases implemented using multiple clusters, according to some embodiments.

[0004] Figure 2 This is a flowchart illustrating a method for calculating the failure rate of one or more clusters in a database, according to some embodiments.

[0005] Figure 3 This is a block diagram illustrating example health information stored in a database cluster, according to some embodiments.

[0006] Figure 4 The flowchart illustrates a method for generating transformation decisions for a database implemented using multiple clusters, according to some embodiments.

[0007] Figure 5 This is a flowchart illustrating a method for generating health information that can be used to determine whether to switch from accessing the active cluster to accessing the backup cluster, according to some embodiments.

[0008] Figure 6 The diagram illustrates an example computing device according to some embodiments. Detailed Implementation

[0009] To synchronize database clusters, the health status of these clusters must be determined. Traditional techniques for identifying cluster health information include implementing an external coordinator, such as the sentinel mechanism used in in-memory data structures (e.g., Redis key-value stores). This sentinel mechanism periodically sends heartbeats to each cluster of the database. Another example of an external coordinator is Apache's ZooKeeper. The heartbeats sent by the external coordinator are simple pulse signals used to check the health status of the clusters in the database. For example, these heartbeats indicate whether the database cluster is active. Based on this information, the sentinel votes on whether a given cluster is healthy. However, this type of health check does not reflect real-time communication traffic between the host computer and the database. For example, in some cases, the database cluster may be active but unable to properly serve requests from client computers. In this case, the sentinel will report the database cluster as healthy even if client requests to the cluster fail or the cluster cannot provide sufficient response time (and may therefore time out).

[0010] Techniques for providing a consistent health voting system that does not involve an external coordinator are disclosed. Specifically, the techniques disclosed herein allow access host computers in a pool to vote for a cluster in a non-relational database based on real-time traffic between client host computers and the database, rather than simple active / inactive pulse checks (e.g., heartbeats). Note that the access host computers provide access to the database to one or more client host computers by maintaining connections to the database. Because the disclosed health voting system is built within a database system layer (which provides access to the database that maintains the pool of access host computers (e.g., a database access layer), monitoring real-world traffic is possible. For example, a particular access host computer may attempt multiple database operations on data stored in the active cluster in the database. Based on the results of these database operations (e.g., the number of failures, timeouts, and successes), the access host computers vote, which specifies the health status of the active cluster. The system of this disclosure then makes a transition decision for that particular cluster based on the votes from multiple host computers. For example, if a vote from a majority of the accessing host computers included in the pool indicates that the active cluster is an unhealthy cluster, the system of this disclosure will store a transition decision in the database for each cluster, indicating a switch from the active (unhealthy) cluster to the standby (healthy) cluster. Storing the transition decision in both the active and standby clusters in the database allows the host computers to access the transition decision even if only one of these clusters is active.

[0011] The techniques disclosed herein can advantageously provide high availability to various client host computers attempting to access a distributed database, while preventing inconsistencies across different clusters of the distributed database. As discussed above, a high-availability database can be a database designed to maintain a predetermined level of operational performance (e.g., the database is generally available to computer systems attempting to access it). Furthermore, preventing inconsistencies between clusters by providing accurate and consistent data to the requesting system can improve the overall performance of the database system. Moreover, storing the health information of the distributed database across multiple clusters ensures that transition decisions are available to the host computer even in the event of a failure of one or more database clusters. For example, if the active cluster is currently failing, the host computer will be able to access transition decisions stored in the standby cluster, and as a result, the host computer will know to switch to using the standby cluster instead of the active cluster. Because the transition decisions generated using the techniques disclosed herein are based on real-time database traffic, they provide a more accurate representation of the health status of the clusters in the distributed database compared to conventional health status techniques (e.g., those using heartbeats).

[0012] Example Database System

[0013] Figure 1 This is a block diagram illustrating an example database system configured to manage databases implemented using multiple clusters. In the illustrated embodiment, system 100 includes a computer system 110, one or more indirect client services 180 (which in turn include client host computers 182A to 182N), and a database 160 (which in turn includes an active database cluster 130A and a backup database cluster 130B). In some embodiments, system 100 includes one or more direct client services 170.

[0014] In the illustrated embodiment, computer system 110 communicates with various client host computers 182 included in one or more indirect services 180 to provide access to data stored in database 160. Computer system 110 may receive various access requests 184 from one or more client host computers 182 included in one or more indirect client services 180. These requests are database queries. Based on these requests, computer system 110 may use various database connections to propagate database operations to database 160, and these database connections are maintained by access host computers 122 included in a pool of access host computers. In some embodiments, computer system 110 is a data access service acting as a data access layer (DAL). Client services (e.g., service 180) connect to the data access service to access database 160. Note that client host computers 182 submit requests to access database 160, while access host computers 122 maintain connections to the database to provide access to client host computers 182. Computer system 110 maintains a pool of access host computers 122 based on the number of database connections required to satisfy database access requests received from various client services. This pool can include hundreds of access host computers that provide data access services to various client services. Computer system 110 can be referred to as a distributed data access service.

[0015] As used herein, the term "data access layer" is intended to be interpreted according to its fully understood meaning, encompassing computer software that provides simplified access to various types of data stored in persistent storage (e.g., databases). In some embodiments, the data access layer provides access to various non-relational databases. For example, the data access layer may provide access to databases other than Structured Query Language (NoSQL). In other embodiments, the data access layer provides access to relational databases. The techniques of this disclosure can be implemented for any of various types of databases. References are made below. Figure 2 The computer system 110 discussed may be a gateway device that provides access to a database cluster included in the database 160 to various client host computers. In some embodiments, one or more direct client services 170, including various client host computers 172, may directly access the database 160 (i.e., without connecting to the computer system 110).

[0016] Indirect client services 180 may include multiple client host computers 182A to 182N that access database 160 via computer system 110. These services(one or more) 180 may include various virtual host computers that access the same set of physical servers and storage resources. In some cases, the client host computers include user interfaces that allow individuals to use these computers for various client applications (e.g., accessing data stored in database 160).

[0017] In the illustrated embodiment, database 160 includes two distinct database clusters storing replicated datasets: an active database cluster 130A and a backup database cluster 130B. In some embodiments, database 160 is a non-relational database. In some embodiments, database 160 includes a set of active database clusters and a set of backup database clusters. As used herein, the term "database cluster" is intended to be interpreted according to its meaning as fully understood in the context of a database system comprising a collection of database nodes. These database nodes may be replicas of each other, in which case each node acts as a backup of the data stored on the other nodes. In some embodiments, database clusters 130A and 130B are implemented in Aerospike, an in-memory open-source NoSQL database.

[0018] Database 160 resides in a data center, which may be a facility comprising various hardware storage systems for storing, processing, and distributing data to different computer systems included in a given computer network. In some embodiments, multiple data centers at different physical locations may store replicated data of an entity (e.g., an enterprise). For example, computer system 110 may have easy access to a data center in Salt Lake City, Utah, while another computer system has easy access to a data center in Las Vegas, Nevada. In addition to the redundancy provided by multiple data centers distributed across different physical locations, the implementation of two clusters 130 within a given data center provides the redundancy required for a high-availability (HA) database. As discussed above, in various situations, active cluster 130A may become out of sync with backup cluster 130B, resulting in inconsistencies between the data stored in the two clusters. The techniques of this disclosure provide a consistent high-availability mechanism to ensure that clusters on data centers have identical data. For example, in some embodiments, cluster 130A is located in a Utah data center, while cluster 130B is located in a Nevada data center.

[0019] To maintain consistency between the two database clusters 130 implemented within database 160, computer system 110 collects votes 102A to 102N from accessing host computer 122 and determines whether one or more clusters in database 160 currently exhibit inferior characteristics (e.g., unhealthy due to failure). Such votes can be provided by accessing host computer 122 based on real-time database traffic. For example, client host computer 182 sends a request to computer system 110 to access information stored in database 160. Accessing host computer 122 accesses the clusters in database 160 and provides responses to client host computer 182 accordingly. Based on the responses to these requests (e.g., success, failure, timeout, etc.), accessing host computer 122 generates health votes specifying whether the individual clusters in the database are healthy.

[0020] Once a cluster is determined to be unhealthy, it is no longer accessed. An unhealthy cluster can be one that is no longer synchronized with its replication cluster (e.g., database cluster 130A no longer contains the same data as database cluster 130B). Under normal operating conditions, the corresponding clusters stored in the database (e.g., clusters 130A and 130B) are intended to store the same information. In database systems where cluster health information is unavailable or not maintained (e.g., there is no sentinel to check cluster health), a database cluster may become unhealthy because different accessing host computers access both the active and backup clusters (instead of all host computers accessing the same cluster, such as active cluster 130A, within a given time frame). Health monitoring systems using sentinels attempt to mitigate this problem, for example, by periodically sending pulses to the database to determine if the cluster is still healthy. Based on these checks, the sentinel can provide health information instructing all host computers to access a set of healthy clusters. However, such sentinels do not provide health information based on real-time database traffic, which can lead to inaccurate information provided by these sentinels. The health selection technique disclosed herein attempts to address this problem by determining whether to switch to a backup cluster based on real-time database traffic.

[0021] Computer system 110 generates health information 140 based on votes 102 collected from various access host computers 122. These votes may be referred to herein as health votes or ballots. Computer system 110 collects votes from access host computers 122 and generates a conversion decision. For example, the decision may be based on a majority of votes. Specifically, when a majority (i.e., majority) votes are available (a majority of access host computers have provided health votes), computer system 110 generates a conversion decision based on these votes. For example, a majority may include at least 50% of the access host computers maintained by computer system 110. Computer system 110 then stores copies of this decision 144 along with the individual health votes 142 in active cluster 130A and backup cluster 130B.

[0022] The transition decision 144 specifies whether the host computer should switch from accessing cluster 130A to accessing backup cluster 130B. This transition decision 144 can also be referred to as a failover decision. In the event of a transition decision, access to the active cluster currently serving live traffic will cease, and all live traffic will be switched to the backup (or standby) cluster. The active cluster can be referred to as the primary cluster, and the backup cluster can be referred to as the local data recovery cluster. After the access transition decision 144, the access host computer 122 implements a switchover action thread. This thread is the smallest sequence of programming instructions that can be independently managed by the computer's operating system. This switchover action thread executes the transition when the transition decision 144 instructs the active cluster to stop service. See below for reference. Figure 3 Let's discuss this switching action thread in more detail.

[0023] The storage of health information 140 in both the active and backup clusters of the database provides an advantageous guarantee that the information remains available even if one cluster becomes unavailable. As discussed above, generating health information based on real-world database traffic provides a more accurate representation of the health of various database clusters. Consequently, transformation decisions generated based on this health information will accurately reflect the current state of the clusters in the database. Because the database health system of this disclosure is implemented via two independent database clusters (e.g., an active cluster and a backup cluster), no additional infrastructure or external coordinator (e.g., ZooKeeper) is required, thus providing an easily executable and maintainable system.

[0024] As described above, in some cases, various direct client services 170 can also access data stored in database 160. These services(one or more) 170 can directly access database 160. That is, while indirect client services 180 access database 160 via computer system 110, client services 170 directly access the database. Although services(one or more) 170 may include various client host computers 172, the number of host computers included in these services is typically negligible compared to the number of host computers included in indirect services(one or more) 180. Therefore, host computers 172 included in one or more services 170 do not participate in voting on the health status of the cluster implemented in the database. These client services(one or more) 170 simply read the transformation decision 144 directly from database 160 and perform future database accesses based on the transformation decision. For example, services(one or more) 170 change which cluster they access for future queries based on this decision. Therefore, client services are divided into two categories: those client services that access database 160 via the data access layer (i.e., computer system 110) and those client services that directly access the database. The former participates in the healthy voting process disclosed herein, and the data access layer handles the generation of ballots and conversion decisions in a manner transparent to these client services, while the latter does not participate in the voting process and only reads the conversion decision from the database after it is generated.

[0025] Example Failure Rate Calculation

[0026] Figure 2 This is a flowchart illustrating a method 200 for performing health checks on one or more clusters in a database. Figure 2 The methods illustrated herein can be used in conjunction with the computer circuits, systems, devices, components, or assemblies disclosed herein, as well as other devices. For example, computer system 110 can perform... Figure 2 The methods shown are illustrated. In various embodiments, some of the method elements shown may be executed simultaneously, in a different order than that shown, or may be omitted. Additional method elements may also be executed as needed.

[0027] At element 202, the method begins. At element 210, the computer system determines whether the database system is ready. For example, if an indicator indicating the health of a set of database clusters (e.g., a set of active clusters) has been removed, the computer system can automatically determine that the current cluster should no longer be accessed (e.g., unhealthy). However, if the indicator specifies that the cluster is ready for access (e.g., healthy), then at 220, the computer system initiates a failure rate calculation for each cluster in the database (e.g., database 160). For example, the failure rate can be calculated for both active and backup clusters, regardless of which cluster is currently being accessed.

[0028] A system readiness indicator is a flag in the database whose value can be manually set or completely removed. In the system of this disclosure, this flag is a record that can be inserted into the database cluster. Based on this record present in the cluster, the computer system 110 determines the value of the record during a health check before taking further action. This flag provides the ability to manually switch to the backup cluster even if the active cluster is still healthy (according to a health check performed by the computer system 110). This implementation can be used to perform maintenance on the active cluster, such as upgrading software versions or replacing outdated hardware. After performing such operations, the system may need to restart the active cluster, and therefore the cluster should not be accessed during the restart. To perform such operations and restart the active cluster, the system of this disclosure, for example, removes the system readiness flag from the active cluster, causing the computer system 110 to generate a transition decision indicating a switch from accessing the active cluster to accessing the backup cluster.

[0029] Each access host computer 122 can calculate the failure rate for each cluster in the database by attempting to access the individual clusters. Based on these attempts indicating failure, success, or timeout for a given cluster, the given access host computer 122 divides the number of timeout and failure responses for that cluster by the total number of access attempts. Success indicators for these access attempts are provided in real time based on the availability of the data stored in database 160. For example, a timeout can indicate that access to a given database cluster took longer than the allowed processing time.

[0030] If the calculated failure rate exceeds a certain threshold, a given accessing host computer may vote the cluster as unhealthy. For example, if a larger number of accesses to a given cluster return failures instead of successes, the cluster may be considered unhealthy. Note that each accessing host computer can calculate failure rates and, based on these rates, vote the clusters in the database as healthy. For example, accessing host computer 122A can vote the cluster as healthy for each of the database clusters 130A and 130B based on information queried from database clusters 130A and 130B. Health votes are also cast based on accesses to the database within a given time interval. For example, a health check process 200 may be initiated every two minutes, providing accessing host computers with ample time to submit multiple queries to the database to access data stored in the various clusters within the database.

[0031] At element 230, the computer system determines whether the number of unhealthy votes from each access host computer to the active database cluster meets a health threshold. For example, if the number of unhealthy votes in the active cluster is greater than or equal to 50% of the total number of votes received from the access host computers, the computer system generates a transition decision indicating that the active cluster is unhealthy. Then, at element 240, the transition decision is stored by the computer system in both the active cluster and the standby cluster. If the number of unhealthy votes in the active cluster does not meet the health threshold, the transition decision is healthy, and the method continues to element 250. In some embodiments, the transition decision is updated to indicate health. For example, in the case where a previous health check result was an unhealthy active cluster, a current health check indicating health can cause the transition decision to be updated accordingly.

[0032] At point 250, the computer system determines whether a purge time threshold has been met. The purge time threshold specifies the amount of time that may have elapsed before the system of this disclosure removes outdated votes from the health information. Votes can be considered outdated when the access host computer associated with them is no longer active (e.g., the database connections associated with these computers are no longer maintained in the pool). This determination is made by subtracting the timestamp of the vote generated by the given access host computer and stored in the database from the current time. If this value is greater than the purge time threshold, the computer system removes the access host computer's votes from the health information.

[0033] The predetermined hibernation interval can be selected based on the volume of database traffic. For example, if the selected interval is too short, too little database traffic may be observed during that interval, and therefore the failure rate calculated based on that traffic may not accurately reflect the health of the database cluster (e.g., the rate may be affected by some random factor such as network jitter). Conversely, if the selected interval is too long, the vote collection used to determine the transition decision may take longer than expected. In some instances, the purge time threshold can be twice the predetermined hibernation interval. That is, the computer system may undergo more than three iterations of health checks before determining whether the access host computer has been removed from pool 120 (and therefore no longer participates in health voting). This ensures that outdated votes are not considered when generating the transition decision.

[0034] At point 252, the computer system waits for a predetermined time interval before resuming the health check process. For example, this interval could be 60 seconds. For the database, health checks 200 can be performed periodically to ensure that the appropriate clusters within the database are being accessed.

[0035] Note that the various examples in this document generate transformation decisions for example non-relational databases, but these examples are discussed for illustrative purposes and not intended to limit the scope of this disclosure. In other embodiments, any of various types of databases may be implemented.

[0036] Example of health information storage

[0037] Figure 3 This is a block diagram illustrating example health information stored in a cluster of databases. In the illustrated embodiment, system 300 includes computer system 110 and database 160, which in turn includes an active cluster and a backup cluster 330, which have [reference to above] Figure 2 The discussed example health check method generates copies of the voting results. In some embodiments, the copies of the voting results stored in the active cluster and backup cluster 330 include voting results generated based on health votes provided by various access host computers 122 of the access database 160.

[0038] In the illustrated embodiment, computer system 110 stores health votes 302 for the active cluster and backup cluster in each cluster of database 160. For example, copies of the voting results 310A of the active cluster and 320A of the backup cluster are stored in the active cluster 330A, while copies of the voting results 310B of the active cluster and 320B of the backup cluster are also stored in the backup cluster 330B. This storage configuration provides access to health information even if either the active cluster or the backup cluster is unavailable.

[0039] In the illustrated embodiment, an enlarged version of the voting results 320A for backup cluster 330B is shown. These voting results are stored in a key-value configuration, where the hostname of each host computer is stored in a key field 402, and a timestamp (e.g., the time the vote was cast) and health votes (e.g., binary values ​​where 0 is unhealthy and 1 is healthy) are stored in a value field 404 for the active cluster in the database. Thus, the latest vote cast by a given accessing host computer for the backup cluster is stored in the database as a key-value pair. That is, a vote cast by a given accessing host computer can indicate whether a majority of votes specify "healthy" for the active cluster. Voting results 310A and 310B for active cluster 330A provided by the accessing host computers are stored in database 160 in a similar manner. In other embodiments, votes cast by a given accessing host computer indicate whether a particular cluster is healthy or unhealthy. Each accessing host computer can cast a vote for each cluster in both the active and backup clusters, and these votes are then stored in database 160 as key-value pairs.

[0040] In some embodiments, access host computers 122, included in a pool within computer system 110, periodically read transition decisions 144 from database 160. For example, every 10 seconds, a switch action thread may be executed by each host computer to determine whether an action is needed to change from accessing the active cluster to accessing the backup cluster, and vice versa. This switch action thread is a lightweight thread that runs on short cycles to allow the systems of this disclosure to react quickly to transition decisions. In the 10-second example, the switch action thread would check the transition decision every 10 seconds and take action accordingly. Therefore, a host computer will take action 10 seconds after the transition decision is stored in the database cluster.

[0041] Example Method

[0042] Figure 4 The flowchart illustrates a method for generating transformation decisions for a database implemented using multiple cluster sets, according to some embodiments. Figure 4 The methods illustrated herein can be used in conjunction with the computer circuits, systems, devices, components, or assemblies disclosed herein, as well as other devices. For example, computer system 110 can perform... Figure 4 The methods shown are illustrated. In various embodiments, some of the method elements shown may be executed simultaneously, in a different order than that shown, or may be omitted. Additional method elements may also be executed as needed.

[0043] At 410, in the illustrated embodiment, the computer system receives multiple requests to access data from multiple host computers, the data being stored in a database implemented using multiple clusters. In some embodiments, the database is a non-relational database. In some embodiments, the active cluster and backup cluster include redundant data. In some embodiments, the computer system acts as a gatekeeper for the non-relational database, facilitating communication between the multiple host computers and the database. In some embodiments, the computer system receives requests from one or more client host computers included in a client service. For example, computer system 110 may receive a request to access database 160 from client host computer 182 in one or more services 180.

[0044] At 420, the computer system responds to multiple requests by accessing data stored in the active cluster. In some embodiments, the computer system uses one or more access host computers maintained in a pool of host computers to access the data. These access host computers maintain database connections.

[0045] At 430, the computer system determines health information for a cluster among multiple clusters based on the response, wherein the health information is generated based on real-time database traffic. In some embodiments, determining the health information includes determining whether one or more database accesses were successful, and wherein the health information indicates poor characteristics of the active cluster.

[0046] In some embodiments, health information is generated by the accessing host computer by sending multiple requests for data to the database via the computer system. In some embodiments, the host computer then determines a health value for each database access based on the success of the database access. In some embodiments, the host computer then calculates the failure rate of one or more clusters in the database based on this health value. For example, if a given cluster has 3 failures and 1 timeout in 1000 database accesses to that cluster, the failure rate of that cluster would be 4 / 1000. In some embodiments, the generation of health information is also based on determining whether the failure rate calculated for each cluster meets a health threshold, wherein the health information is based on this determination to indicate whether each cluster in the database is healthy. Therefore, for a health threshold of 60%, a cluster with a failure rate of 4 / 1000 would be considered healthy.

[0047] At point 440, the computer system determines whether to switch from accessing the active cluster to accessing the backup cluster based on health information. This determination may be referred to as a transition decision. For example, if more than 50% of the clusters in the active cluster set are unhealthy (based on the failure rate calculated at point 430), the computer system determines that the host computer should switch to accessing the backup cluster instead of the active cluster. In some embodiments, the active cluster set and the backup cluster set include redundant data. In some embodiments, the computer system periodically receives updated health information from access host computers (configured to respond to multiple requests) included in the access host computer pool. The updated health information is generated based on real-time changes in database traffic. In some embodiments, the computer system updates the transition decision based on the updated health information.

[0048] In some embodiments, the computer system periodically determines whether one or more access host computers have been removed from the pool based on a purge time threshold. In some embodiments, the computer system removes one or more health votes associated with an access host computer that has been removed from the pool from the health information. For example, if an access host computer is no longer active in the access host computer pool, old health votes cast by that access host computer should be removed from the pool because these votes have expired. In some embodiments, the purge time threshold is three minutes.

[0049] At 450, the computer system stores the transformation decision generated based on this determination in each cluster of the database. Each access host computer 122 can access the transformation decision to determine which group of clusters in the database to access for a future database request. In some embodiments, the stored transformation decision can be accessed by one or more access host computers to determine whether to access the active cluster or the backup cluster included in the database based on a request received from a host computer among a plurality of host computers. For example, these host computers are client host computers 182 included in client service 180.

[0050] In some embodiments, the database is a key-value database, and the computer system is a distributed microservice that acts as the data access layer (DAL) of the key-value database. In some embodiments, the computer system includes a pool of access host computers configured to maintain multiple connections to the database. In some embodiments, the conversion decision includes health voting on each cluster of the key-value database, and for each cluster in the database, these health votes specify the name of the access host computer as an entry in the key field, and specify a timestamp and health vote as entries in the value field.

[0051] Figure 5The flowchart illustrates, according to some embodiments, a method for generating health information that can be used to determine whether to switch from accessing the active cluster set to accessing the backup cluster set. Figure 5 The methods illustrated herein can be used in conjunction with the computer circuits, systems, devices, components, or other devices disclosed herein. For example, accessing one of the host computers 122 can perform... Figure 5 The methods shown are as follows. In various embodiments, some of the method elements shown may be executed simultaneously, in a different order than that shown, or may be omitted. Additional method elements may also be executed as needed.

[0052] At point 510, in the illustrated embodiment, the host computer accesses data stored in a database implemented using multiple clusters. Multiple requests may include database operations, such as data processing operations.

[0053] At point 520, the host computer receives a response from the database indicating whether access to data stored in the database cluster was successful. These responses may include indications of success, failure, or timeout. Furthermore, these responses may include data retrieved from the active cluster of the database.

[0054] At point 530, the access host computer determines a health value for each database access based on the response. This health value can be used by the gateway device to generate health information for clusters in multiple clusters. This health information can be used by one or more access host computers maintained by the gateway device to determine whether to switch from the active cluster of the access database to a backup cluster of the access database. In some embodiments, this determination includes calculating a failure rate for one or more clusters in the database based on the health value. In some embodiments, calculating the failure rate includes dividing the total number of timeouts and failed health values ​​by the total number of determined health values.

[0055] In some embodiments, access to the host computer is determined by a transition decision stored by the gateway device, wherein the transition decision is generated by the gateway device based on health information. In some embodiments, the host computer switches database access from the active cluster implemented by the database to the backup cluster implemented by the database based on the transition decision.

[0056] Example computing device

[0057] Turn now Figure 6 , Figure 6This is a block diagram illustrating one embodiment of a computing device (which may also be referred to as a computing system) 610. The computing device 610 can be used to implement various parts of this disclosure. The computing device 610 can be any suitable type of device, including but not limited to personal computer systems, desktop computers, laptop or notebook computers, mainframe computer systems, network servers, workstations, or network computers. As shown, the computing device 610 includes a processing unit 650, a storage device 612, and an input / output (I / O) interface 630 coupled via an interconnect 660 (e.g., a system bus). The I / O interface 630 can be coupled to one or more I / O devices 640. The computing device 610 also includes a network interface 632, which can be coupled to a network 620 for communicating with, for example, other computing devices.

[0058] In various embodiments, processing unit 650 includes one or more processors. In some embodiments, processing unit 650 includes one or more coprocessor units. In some embodiments, multiple instances of processing unit 650 may be coupled to interconnect 660. Processing unit 650 (or each processor within 650) may include cache or other forms of on-board memory. In some embodiments, processing unit 650 may be implemented as a general-purpose processing unit, while in other embodiments, processing unit 650 may be implemented as a special-purpose processing unit (e.g., ASIC). Generally, computing device 610 is not limited to any particular type of processing unit or processor subsystem.

[0059] The storage subsystem 612 may be used by the processing unit 650 (e.g., for storing instructions that can be executed by the processing unit 650 and data that can be used by the processing unit 650). The storage subsystem 612 may be implemented by any suitable type of physical memory medium, including hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDORAM, SDRAM, DDR SDRAM, RDRAM, etc.), ROM (PROM, EEPROM, etc.), etc. In one embodiment, the storage subsystem 612 may consist only of volatile memory. The storage subsystem 612 may store program instructions that can be executed by the computing device 610 using the processing unit 650, including program instructions executable to enable the computing device 610 to implement the various techniques disclosed herein.

[0060] According to various embodiments, I / O interface 630 may represent one or more interfaces and may be any type of interface configured to couple to and communicate with other devices. In one embodiment, I / O interface 630 is a bridge chip from a front end to one or more back end buses. I / O interface 630 may be coupled to one or more I / O devices 640 via one or more corresponding buses or other interfaces. Examples of I / O devices include storage devices (hard disks, optical drives, removable flash drives, storage arrays, SANs, or associated controllers), network interface devices, user interface devices, or other devices (e.g., graphics, sound, etc.).

[0061] Various articles of art that store instructions (and optionally data) executable by a computing system to implement the techniques disclosed herein are also considered. The computing system may use one or more processing elements to execute the instructions. The articles of art include non-transitory computer-readable storage media. The non-transitory computer-readable storage media considered include portions of the memory subsystem of a computing device and storage media or memory media such as magnetic media (e.g., magnetic disks) or optical media (e.g., CDs, DVDs, and related technologies). Non-transitory computer-readable media can be volatile or non-volatile memory.

[0062] This disclosure includes references to “exemplary embodiments,” which are non-limiting implementations of the disclosed concepts. References to “exemplary,” “an embodiment,” “a particular embodiment,” “some embodiments,” “various embodiments,” etc., do not necessarily refer to the same embodiment. A large number of possible embodiments are contemplated, including specific embodiments described in detail, as well as modifications or substitutions falling within the spirit or scope of this disclosure. Not all embodiments are required to exhibit any or all of the potential advantages described herein.

[0063] Unless otherwise stated, the specific embodiments are not intended to limit the scope of the claims drafted based on this disclosure to the disclosed form, even if only a single instance is described with respect to a particular feature. Therefore, without any statement to the contrary, the disclosed embodiments are intended to be illustrative rather than restrictive. This application is intended to cover such alternatives, modifications, and equivalents that will be apparent to those skilled in the art benefiting from this disclosure.

[0064] Specific features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure. Therefore, this disclosure is intended to include any feature or combination of features disclosed herein (explicitly or implicitly) or any generalization thereof. Consequently, new claims may be formulated during the examination of any such combination of features in this application (or an application claiming priority thereto). Specifically, referring to the appended claims, features from dependent claims may be combined with features from independent claims, and features from corresponding independent claims may be combined in any suitable manner, not merely in the specific combinations listed in the appended claims.

[0065] For example, while the appended dependent claims are drafted such that each dependent claim depends on a single other claim, additional dependent claims are also considered, including: claim 3 (which may be dependent on any one of claims 1 to 2), claim 4 (which may be dependent on any of the preceding claims), claim 5 (which may be dependent on claim 4), etc., and where appropriate, claims drafted in one statutory type (e.g., apparatus) may also suggest corresponding claims in another statutory type (e.g., method).

[0066] Because this disclosure is a legal document, various terms and phrases are subject to administrative and judicial interpretation. This document draws public attention to the following paragraphs, as well as the definitions provided throughout this disclosure, which will be used to determine how the claims drafted based on this disclosure should be interpreted.

[0067] Unless the context clearly specifies otherwise, references to the singular forms (e.g., “a,” “an,” and “the”) are intended to mean “one or more.” Therefore, a reference to the item “an” in a claim does not exclude additional instances of that item.

[0068] The word “can” is used in this text in a permissive sense (i.e., having the potential, being able to) rather than in a mandatory sense (i.e., being required to).

[0069] The terms "comprising" and "including," and their forms, are open-ended and mean "including but not limited to."

[0070] When the term “or” is used in this disclosure with respect to a list of options, it is generally understood to be used in an inclusive sense, unless the context otherwise specifies. Thus, the expression “x or y” is equivalent to “x or y or both,” which covers x but not y, covers y but not x, and covers both x and y. In this respect, phrases such as “x or y, but not both” clearly indicate that “or” is used in an exclusive sense.

[0071] The phrases “w, x, y, or z or any combination thereof” or “...at least one of w, x, y, and z” are intended to cover all possibilities involving a single element to the total number of elements in the set. For example, given the set [w, x, y, z], these phrases cover any single element of the set (e.g., w but not x, y, or z), any two elements (e.g., w and x but not y or z), any three elements (e.g., w, x, and y but not z), and all four elements. Therefore, the phrase “...at least one of w, x, y, and z” refers to at least one element in the set [w, x, y, z], thus covering all possible combinations in the list of options. This phrase should not be interpreted as requiring the existence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0072] Various “labels” may continue to be used as terms in this disclosure. Unless the context otherwise requires, different labels used for a feature (e.g., “first circuit”, “second circuit”, “specific circuit”, “given circuit”, etc.) refer to different instances of that feature. Unless otherwise stated, the labels “first,” “second,” and “third” do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) when applied to a particular feature.

[0073] In this disclosure, different entities (which may be referred to differently as “units,” “circuits,” other components, etc.) may be described or declared as “configured” to perform one or more tasks or operations. This expression ([entity] configured to [perform one or more tasks]) is used herein to refer to a structure (i.e., something of an entity). More specifically, this expression is used to indicate that the structure is set to perform one or more tasks during operation. Even if the structure is not currently being operated, it can be said that the structure is “configured” to perform some tasks. Therefore, an entity described or stated as “configured to” perform some tasks refers to something of an entity, such as a device, circuit, memory storing program instructions, etc., which can be executed to perform the task. This phrase is not used herein to refer to something abstract.

[0074] The term "configured as" is not intended to mean "configurable as". For example, an unprogrammed FPGA would not be considered "configured as" to perform some specific function. However, the unprogrammed FPGA could be "configurable as" to perform that function.

[0075] The appended claims state that the structure is “configured” to perform one or more tasks, a statement explicitly intended not to invoke 35 U.SC § 112(f) for that claim element. If the applicant wishes to invoke 112(f) during examination, it will use the construction “means for [performing functions]” to state the claim element.

[0076] The phrase "based on" is used to describe one or more factors that influence a determination. This term does not exclude the possibility that other factors might influence the determination. That is, a determination can be based solely on the specified factors or on the specified factors along with other unspecified factors. Consider the phrase "A is determined based on B." This phrase specifies that B is a factor used to determine A or that influences the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on other factors, such as C. This phrase is also intended to cover embodiments in which A is determined solely based on B. As used herein, the phrase "based on" is synonymous with the phrase "at least partially based on."

[0077] The phrase "in response to" describes one or more factors that trigger an effect. This phrase does not exclude the possibility that other factors might influence or otherwise trigger the effect. That is, an effect may respond only to those factors, or it may respond to the specified factors as well as other unspecified factors. Consider the phrase "execute A in response to B." This phrase specifies that B is the factor that triggers the performance of A. This phrase does not exclude that the execution of A may also respond to some other factors, such as C. This phrase is also intended to cover embodiments in which A is executed only in response to B.

Claims

1. A method comprising: The computer system receives multiple requests to access data from multiple host computers, the data being stored in a database implemented using multiple clusters; The computer system responds to the multiple requests by accessing data stored in the active cluster; The computer system determines the health information of one of the plurality of clusters based on the response indicating whether access to the data was successful, wherein the health information is generated based on the real-time traffic of the database; The computer system periodically determines, based on a clearing time threshold, whether one or more access host computers have been removed from the pool of access host computers used to respond to the plurality of requests; The computer system removes one or more health votes associated with one or more access host computers that have been removed from the pool from the health information; The computer system determines whether to switch from accessing the active cluster to accessing the backup cluster based on the health information; and The computer system stores the transformation decisions generated based on the determination in each cluster of the database.

2. The method according to claim 1, wherein, The database is a non-relational database, wherein the active cluster and the backup cluster include redundant data, and wherein the computer system is the gatekeeper of the non-relational database, enabling communication between the multiple host computers and the database.

3. The method according to claim 1, wherein, Determining the health information includes: determining whether one or more database accesses were successful, and wherein the health information indicates poor characteristics of the active cluster.

4. The method according to claim 1, further comprising: The computer system periodically receives updated health information from access host computers included in a pool of access host computers, the access host computers being configured to respond to the plurality of requests, wherein the updated health information is generated based on real-time traffic changes in the database; and The conversion decision is updated based on the updated health information.

5. The method according to claim 1, wherein, The health information is generated by the accessing host computer based on the following operations: Multiple requests for data are sent to the database via the computer system; The health value of each database access is determined based on the success of the database access. as well as Based on the health value, calculate the failure rate of one or more clusters in the database.

6. The method according to claim 5, wherein, The generation of the health information is also based on: Determine whether the failure rate meets a health threshold, wherein the health information is based on the determination to indicate whether one or more clusters in the database are healthy.

7. The method according to claim 1, wherein, The stored transformation decisions can be accessed by one or more access host computers to determine whether to access the active cluster or the backup cluster included in the database based on a request received from one of the host computers.

8. The method according to claim 1, wherein, The database is a key-value database, wherein the computer system is a distributed microservice that acts as the data access layer (DAL) of the key-value database, and wherein the computer system includes a pool of access host computers that maintain multiple connections to the database.

9. The method according to claim 8, wherein, The transformation decision includes a health vote on the clusters of the key-value database, wherein, for each cluster in the database, the health vote specifies: Access the host computer's name as an entry in the key field; and Timestamps and health votes are entries in the value field.

10. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by a gatekeeper device to perform operations, said operations including: Receive multiple requests to access data from multiple host computers, wherein the data is stored in a database implemented using multiple clusters; The multiple requests are responded to by accessing data stored in the active cluster; The health information of the clusters in the plurality of clusters of the database is determined based on whether the access to the data is successful, wherein the health information is generated based on the real-time traffic of the database; Based on the health information, determine whether to switch from accessing the active cluster to accessing the backup cluster; The transformation decisions generated based on the determination are stored in each cluster of the database; Periodically receive updated health information from access host computers included in a pool of access host computers, the access host computers being configured to respond to the plurality of requests, wherein the updated health information is generated based on real-time traffic changes to the database; and The conversion decision is updated based on the updated health information.

11. The non-transitory computer-readable medium according to claim 10, wherein, The database is a key-value database, and the gatekeeper device is a distributed microservice that acts as the data access layer (DAL) of the key-value database.

12. The non-transitory computer-readable medium according to claim 11, wherein, The transformation decision includes a health vote on the clusters of the key-value database, wherein, for each cluster in the database, the health vote specifies: Access the host computer's name as an entry in the key field; and Timestamps and health votes are entries in the value field.

13. The non-transitory computer-readable medium according to claim 10, wherein, The operation also includes: Based on a clearing time threshold, it is periodically determined whether one or more host computers are no longer active; and Based on the determination, information associated with the one or more computers that are no longer active is removed from the health information.

14. The non-transitory computer-readable medium according to claim 10, wherein, The health information is generated by the accessing host computer based on the following operations: Send multiple requests for data to the database; The health value of each database access is determined based on the success of the database access. as well as Based on the health value, calculate the failure rate of one or more clusters in the database.

15. A method comprising: Access data of access host computers in a pool of access host computers maintained by a gateway device, wherein the data is stored in a database implemented using multiple clusters; The accessing host computer receives a response from the database, the response indicating whether access to data stored in the cluster of the database was successful; The access host computer determines a health value for each database access based on the response, wherein the health value can be used by the gateway device to generate health information for clusters among the plurality of clusters, and wherein the health information can be used by one or more access host computers in the pool of access host computers to determine whether to switch from the active cluster accessing the database to a backup cluster accessing the database; and The failure rate of one or more clusters in the database is calculated based on the health value.

16. The method of claim 15, further comprising: The accessing host computer accesses a conversion decision stored by the gateway device, wherein the conversion decision is generated by the gateway device based on the health information.

17. The method of claim 16, further comprising: Based on the conversion decision, the access host computer switches database access from the active cluster to the backup cluster implemented by the database.

18. The method of claim 16, further comprising: The access host computer periodically sends updated health values ​​to the gateway device, wherein the updated health values ​​are generated based on real-time traffic changes in the database, and wherein the gateway device is configured to update the conversion decision based on the updated health values.

19. The method according to claim 18, wherein, Calculating the failure rate involves dividing the total number of timeouts and fault health values ​​by the total number of determined health values.

Images