Switching to the last consistent database replica

Abstract

The present disclosure provides consistent reads in a global database that is replicated to one or more global clones. A state machine is implemented and executed independently for each item in each region. Based on the state of the item when a read request is received, the read request will be serviced by the global database, serviced by a global clone, or ignored as an invalid request.

Description

[0001] Cross-referencing of related applications

[0002] This application is a continuation of U.S. Patent Application No. 16 / 682,513, filed November 13, 2019, the disclosure of which is incorporated herein by reference. Background Technology

[0003] Infrastructure as a Service (IaaS) is supported by a global network of computing devices that serve as the control and data plane. Data storage provides data for such large control plane services. Multiple distinct regions exist within the global network. While some computing resources are isolated within a region, others are global. However, the control plane typically does not communicate directly with the global database. A global stack is maintained for global resources. For example, a stack can be a set of computing resources with business logic. For instance, when a client wants to create a virtual machine, it sends a request to the regional stack, and the business logic implemented in the regional stack probes the global database to retrieve the requested information.

[0004] One problem with the above setup is that all regions are interconnected via network links, and these links can be interrupted. Therefore, the global database is replicated in the global clone database. However, because the global database and the global clone are updated asynchronously, there is typically a delay between updating the global database and replicating such updates in the global clone database. For example, the delay can range from tens of milliseconds to tens of seconds. Therefore, reading from the global database and the global clone database at a given time can result in inconsistent reads. This can trigger multiple problems and side effects in business logic. As an example, if the control plane stores data at a first timestamp and then reads it at a second consecutive timestamp, the data from the first timestamp will always be older than the second timestamp, or it may be the same as the second timestamp. This invariant ensures integrity. For example, a virtual machine resource that has been "deleted" cannot be "undeleted." However, reading from the global database and then sequentially reading from the global clone typically does not provide the same invariant. Instead, it will allow reading a "deleted" virtual machine from the global database and then reading the clone at a slightly later time and finding it still "active." Summary of the Invention

[0005] This disclosure provides consistent reads in a global database replicated to one or more global clones. A state machine is implemented and executed independently for each project within each region. For example, a project could be a set of resources belonging to the same customer. Based on the project's state when a read request is received, the read request may be serviced by the global database, serviced by the global clone, or ignored as an invalid request.

[0006] One aspect of the disclosure provides a method comprising enabling a global clone for a project in a first region, wherein enabling the global clone comprises receiving data for the project from a global database at the global clone for replication. The method further comprises setting, by one or more processors in the first region, a status of the project to a pending status; recording, by the one or more processors in the first region, a timestamp corresponding to a time at which the status was set to the pending status; receiving, by the one or more processors in the first region, a request to read data for the project; and redirecting the request to the global database at a time corresponding to the timestamp when the status of the project is in the pending status. According to some examples, the method can further comprise determining, by the one or more processors in the first region, whether the global clone has been updated with the same data in the global database as of the time of the timestamp; and changing the status of the project to a ready status if the global clone has been updated with the same data.

[0007] It can also include receiving, by the one or more processors in the first region, a second request to read data from the project; and fulfilling the request using data from the global clone when the project is in the ready status. In other examples, it can also include changing the status of the project from the ready status back to the pending status; and recording a second timestamp of a time at which the status changed from the ready status to the pending status.

[0008] Another aspect of the disclosure provides a system comprising a global clone in a first region, the global clone configured to replicate data from a global database of a project; and one or more computing devices. The one or more computing devices are configured to receive data for the project from the global database at the global clone for replication; set a status of the project to a pending status; record a timestamp corresponding to a time at which the status was set to the pending status; receive a request to read data for the project; and redirect the request to the global database at a time corresponding to the timestamp when the status of the project is in the pending status.

[0009] According to some examples, the one or more computing devices can be further configured to determine whether the global clone has been updated with the same data in the global database as of the time of the timestamp; and change the status of the project to a ready status if the global clone has been updated with the same data.

[0010] Yet another aspect of the present disclosure provides a non-transitory computer- readable medium storing instructions executable by one or more processors for performing a method of providing a read request to a global clone. The method includes receiving data for an item from a global database at the global clone for replication, setting a status of the item to a pending status, recording a timestamp corresponding to a time at which the status was set to the pending status, receiving a request to read the data for the item, and redirecting the request to the global database at a time corresponding to the timestamp when the status of the item is in the pending status. According to some examples, execution of the instructions can further provide for determining whether the global clone has been updated with the same data as in the global database as of the time corresponding to the timestamp, and changing the status of the item to a ready status if the global clone has been updated with the same data. Execution of the instructions can also provide for receiving a second request to read the data from the item, and fulfilling the request using the data from the global clone when the item is in the ready status. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 FIGURE illustrates an example system, in accordance with aspects of the present disclosure.

[0012] Figure 2 is a block diagram illustrating example details of components of the system of Figure 1

[0013] Figure 3 is a timing diagram illustrating an example of a delay between a global database and a global clone, in accordance with aspects of the present disclosure.

[0014] Figure 4 is a state diagram illustrating an example of different statuses of an item in a global clone, in accordance with aspects of the present disclosure.

[0015] Figure 5 FIGURE illustrates an example of a data block, in accordance with aspects of the present disclosure.

[0016] Figure 6 is another timing diagram showing an example transaction, in accordance with aspects of the present disclosure.

[0017] Figure 7 is a flow diagram showing an example method of setting a status for an item, in accordance with aspects of the present disclosure.

[0018] Figure 8 is a flow diagram illustrating an example method of servicing a read request based on a status of an item, in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0019] Figure 1 ​An example system 100 is illustrated that includes a global database 110 and multiple regional databases 120, 160. Within each region 105, 155 are partition databases 140, 150, 180, 190. For example, the partitions in which the partition databases 140, 150 reside can be sub-regions of the regions that the regional databases 120 serve. Similarly, the partitions in which the partition databases 180, 190 reside can be sub-regions of the regions that the regional databases 160 serve. These partitions can have high-bandwidth, low-latency network connections to other partitions in the same region. While only a few regions 105, 155 are shown, it should be understood that a global system can include any number of regions, each with any number of partitions.

[0020] Each region 105, 155 also includes a global clone database 130, 170. The global database 110 is asynchronously replicated to the global clone databases 130, 170. In some examples, all data from the global database 110 can be replicated to both clones 130, 170. According to other examples, some data from the global database 110 is replicated to a first clone 130, while other data from the global database 110 is replicated to a second clone 170. For example, a particular project can be replicated to a clone in a particular region.

[0021] Each database is controlled by a stack. For example, a global stack 112 serves as a control plane for the global database 101. Regional stacks 122, 162 serve as control planes for the regional databases 120, 170, respectively. Within the first region 105, partition stacks 142, 152 serve as control planes for the partition databases 140, 150, respectively. Within the second region 155, partition stacks 182, 192 serve as control planes for the partition databases 180, 190, respectively.

[0022] The control plane stores provide a data layer for the control plane services. Each database and stack can have dedicated control plane stores. For example, as shown in Figure 1 the global control plane store 114 provides a data layer for the global stack 112. The regional control plane stores 124, 164 provide data layers for the regional stacks 122, 162, respectively. The partition control plane stores 144, 154, 184, 194 provide data layers for the partition stacks 142, 152, 182, 192, respectively.

[0023] The global database 110 is capable of reading past versions of data. For example, when an update is made to the database 110, rather than immediately erasing the previous information, the previous information can be retained for a period of time. The period of time can be minutes, hours, days, etc. Thus, the global database 110 allows the control plane storage 114 service to read data at any timestamp in the past, such as 30 minutes prior to the "current" timestamp, 4 hours prior, etc.

[0024] Figure 2 Further details are provided regarding the stacks and control plane storage. The regional stack 202 includes a set of computing resources. For example, the stack 202 includes one or more processors 240. The processors 240 are configured to execute data and instructions in the memory 230. For example, the processors 240 can execute the business logic 235 to control the virtual infrastructure (laaS) of the project.

[0025] The processors 240 can be any conventional processor, such as commercially available CPUs. Alternatively, the processors can be a special-purpose component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, the stack 202 can include specialized hardware components to perform particular computing processes.

[0026] The stack 202 can use the database 200 to maintain the state of resources processed by the processors 240. The database 200 storage 230 can be a non-transitory computer-readable medium capable of storing information accessible to the processors 240, such as a hard-drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writeable and read-only memories.

[0027] The memory 230 can store information accessible by the processors 240, including instructions that can be executed by the processors 240 and that can be retrieved, manipulated, or stored by the processors 240.

[0028] The instructions can be sets of instructions directly executable by the processors 240, such as machine code, or indirectly executable, such as scripts. In this respect, the terms "instructions," "steps," and "programs" can be used interchangeably herein. The instructions can be stored in object code format for direct processing by the processors 240, or in other types of computer languages including scripts or collections of independent source code modules, that are interpreted on demand or compiled into machine language prior to processing. The functionality, methods, and routines of the instructions are explained in more detail in the examples above and the example methods below.

[0029] The processor 240 is capable of retrieving, storing, or modifying data according to instructions. The data can also be formatted in a computer-readable format, such as but not limited to binary values, ASCII, or Unicode. Further, the data can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate relevant data.

[0030] The memory 230 includes business logic 235 for controlling data in the database 200. For example, the business logic 235 can determine how to process transactions with the database 200. Depending on the status of the global clone of the project, the business logic 235 will serve read requests differently. For example, and as described further herein, if the global clone of the project is pending, the business logic 235 will read data from the global database at a time corresponding to the timestamp when the clone entered the pending state. If the global clone of the project is ready, the business logic 235 will serve the request by reading data from the global clone.

[0031] Although Figure 2 The processor, computer, computing device, or memory of the computing device 200 is functionally illustrated as being within the same block, the processor, computer, computing device, or memory can actually include a plurality of processors, computers, computing devices, or memories, which can or can not be stored in the same physical housing. For example, the memory can be a hard drive or other storage medium located in a different housing than the computing device 200. Accordingly, references to the processor, computer, computing device, or memory will be understood to include references to a plurality of processors, computers, computing devices, or memories, which can or can not operate in parallel. For example, the computing device 200 can include server computing devices operating as a load-balanced server farm, a distributed system, etc. Further, although some of the functionality described below is indicated as occurring on a single computing device with a single processor, various aspects of the subject matter described herein can be implemented by multiple computing devices, e.g., communicating information over a network.

[0032] The region control plane storage 204 can contain one or more processors 270, memory 260, and other components for providing a data tier to control plane services. The control plane storage 204 can also include one or more state machines 280 for the project that are replicated into the region for the global clone. The state machine 280, for example, can be a computing device programmed to change from one state to another in response to particular inputs or conditions. As described further herein in connection with the examples Figure 4 The states can include an invalid state, a pending state, and a ready state, as described further herein in connection with the examples. In some examples, the states can also include a disabled state.

[0033] Memory 260 is capable of storing information accessible by processor 270, including instructions 262 that can be executed by processor 270. Memory can also include data 264 that can be retrieved, manipulated or stored by processor 270. The memory 260 can be one or more of volatile and non-volatile computer-readable media, such as a hard disk drive, solid-state drive, tape drive, optical storage, flash memory, ROM, RAM, DVD, CD-ROM, writable and read-only memories. The processor 270 can be a well-known processor or other well-known processing device. Alternatively, the processor 270 can be a special purpose controller, such as an ASIC.

[0034] Instructions 262 can be directly executable instructions, such as machine code, or indirectly executable instructions, such as scripts. In this respect, the terms "instructions," "steps," and "programs" can be used interchangeably. Instructions 262 can be stored in object code format for direct processing by processor 270, or in another type of computer language including scripts or collections of independent source code modules, that are interpreted on demand or compiled into machine language prior to processing.

[0035] Processor 270 can retrieve, store, or modify data 264 according to instructions 262. For example, although the system and method is not limited by a particular data structure, data 264 can be stored in computer registers, in a relational database as a table having a plurality of different fields and records, or in an XML document. The data 264 can also be formatted in the computer language such as, but not limited to, binary values, ASCII or Unicode. Moreover, the data 264 can comprise information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information that is used by a function to calculate relevant data.

[0036] Instructions 262 can include performance maintenance activities 267. The maintenance activities can be performed to determine the status of the project in the global clone, and update the state of the global clone of the project based on the determined status. For example, the maintenance activities 267 can determine that the global clone of the project has been enabled, and update the global clone of the project from an invalid state to a pending state. Further, the maintenance activities 267 can determine that the global clone of the project is caught up to a particular point in time of the global database. Accordingly, the maintenance activities 267 can update the state machine 280 of the project to transition the project state from the pending state to the ready state.

[0037] Although Figure 2Processor 270 and memory 260 are shown functionally as being within the same block, but processor 270 and memory 260 can actually comprise a plurality of processors and memory that can or can not be stored in the same physical housing. For example, some instructions 262 and data 264 can be stored on a removable CD-ROM, while others can be stored in read-only memory within a computer chip. Some or all of the instructions and data can be stored on physical media that is physically removed from processor 270 but is still accessible by processor 270. Similarly, processor 270 can actually include a collection of processors that can or can not operate in parallel.

[0038] Database 200 can be a single storage device or multiple storage devices, such as hard drives, random access memory, disks, disk arrays, tape drives, etc. Database 200 can implement any of a variety of architectures and technologies, including but not limited to direct attached storage (DAS), network attached storage (NAS), storage area networks (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), and hybrid architecture networks, etc. Moreover, in some examples, database 200 can include a virtualized or containerized environment. For example, database 200 can include one or more virtual machines running on a host. Database 200 can store, for example, any of data files, documents, code, schemas, persistence frameworks, applications, or various other information or tools typically stored in a database.

[0039] Stack 202 can serve requests from client devices (not shown), such as over a network. The client devices can be configured with processors, memory, instructions, and data. The clients can be personal computers intended for use by one person at a time, which typically have all internal components, such as a central processing unit (CPU), hard disk drive, and input / output devices, as well as all of the components necessary to connect these elements to one another and to a network. The clients can be any device that is capable of processing instructions and transmitting data to and from people and other computers, including general purpose computers, tablet computers, mobile phones, smart watches, home assistants, video game systems, network computers lacking local storage capability, set-top boxes for televisions, and other networked devices.

[0040] Clients can include application interface modules for accessing services provided by the database 200. For example, the application interface modules can include subroutines, data structures, object classes, and other types of software components that allow the server and the clients to communicate with each other. In one aspect, the application interface modules can be software modules that can operate in conjunction with several types of operating systems known in the art. For example, the clients can connect to a structured query language (SQL) database server that can operate with the application interface modules to save and retrieve information data.

[0041] The networks and intermediary nodes can include various configurations and protocols, including the Internet, World Wide Web, intranet(s), virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n, or other such standards), and HTTP and various combinations of the foregoing. Such communication can be facilitated by a device capable of communicating data by wired or wireless communication such as a modem (e.g., dial-up, cable, or fiber-optic) and a wireless interface.

[0042] Figure 3 is a timing diagram illustrating an example of a delay between a global database and a global clone. As shown, version 1 is updated in the global database before time 1. The version can be, for example, a version of a feature, such as a project or application, etc. Due to the delay, version 1 is not replicated in the global clone until after time 2. In this example, by the time version 1 is replicated in the global clone, version 2 is available in the global database. Thus, if a customer request is processed at time 1, the business logic will see version 1 from the global database. At time 2, the business logic will still see version 1 from the global database.

[0043] After time 3, the feature is shipped and enabled in a particular geographic region served by a regional database, such as Figure 1 the first region 105 of FIG. 1. Thus, a customer request processed at time 3 will result in the most recent version from the global database, version 2, because at time 3, the feature has not yet been enabled in the region.

[0044] Before time 4, version 3 is implemented in the global database. However, at this time, version 2 has not yet been implemented in the global clone. Thus, a request processed at time 4 will result in version 1 from the global clone. Although at this time version 3 has been implemented in the global database, the global clone is producing data from the previous version. This can result in unintended side effects. For example, if version 3 is a “deletion” of a virtual network, then producing version 1 from the global clone essentially “undeletes” the virtual network.

[0045] At time 5, the global clone will produce version 2 of the feature, but this version is still without data. It is not until time 6 that the global clone catches up with the global database. Thus, any requests prior to time 6 can receive inaccurate responses.

[0046] The unintended effects caused by the differences at different times between the global database and the global clone can be addressed by implementing a state machine for each item in each region. For example, referring back to Figure 1 , the state machine can be implemented in the region control plane storage 124, 164 of each region 105, 155.

[0047] Figure 4 is a state diagram illustrating examples of different states of an item in the global clone. According to this example, the different states include an invalid state 410, a clone_pending state 420, a clone_ready state 430, and a disabled state 440.

[0048] A new item can be automatically placed in the invalid state 410. In this state, the item can not yet be enabled in the global clone. Thus, all read data requests handled while the item is in the invalid state will produce data from the global database.

[0049] Maintenance activities can discover new items, for example, by scanning. For example, a maintenance activity can read every item resource in the global database and detect whether the item is new. If the item state does not exist in the particular region clone, then the item can be determined to be new. When a new item is detected, the item can be set to the clone_pending state 420. A timestamp of when the item changed from the invalid state 410 to the clone_pending state 420 is recorded. For example, as described further below in connection with Figure 5 , the timestamp can be recorded in the region database along with an indication of the state of the item.

[0050] All global data requests to the region control plane storage of an item in the clone_pending state 420 are redirected to the global database at the time corresponding to the recorded timestamp. For example, an item can transition from the invalid state 410 to the clone_pending state 420 at time ti, and the time ti is recorded. Although a read request can be received at a later time ti+x, if the item is still in the clone_pending state 420, the read request will be redirected to the global database at time ti. Despite the fact that the global database can have been updated with a newer version of the item between time ti and time ti+x, to maintain consistency, the version at time ti will still be produced.

[0051] Maintenance activities in the regional control plane store can also detect transactions for items in the clone_pending state 420. For example, the maintenance activities can detect that a threshold number of reads or writes to the item have occurred. The threshold number can be one or more. When this occurs, the maintenance activities can determine whether the item should transition to the clone_ready state 430. For example, the maintenance activities can verify whether the global clone in the region has caught up to the global database for the item. If the latest version available in the global clone is the same as the version available in the global database as of the record timestamp, then the global clone can be considered caught up. If the global clone is caught up, the maintenance activities change the state of the item to the clone_ready state 430. In the clone_ready state 430, all reads are fulfilled by the global clone.

[0052] The maintenance activities can further detect when an item in the clone_ready state 430 becomes out of sync with the global database. For example, the maintenance activities can periodically check whether the global clone is still caught up to the global database. In other examples, the maintenance activities can detect an update to the global database for the item and, in response, can check whether the update has been replicated to the global clone. When it is determined that the global clone for the item is out of sync with the global database, the state of the item can transition from the clone_ready state 430 back to the clone_pending state.

[0053] At any time, an item can transition to the disabled state 440. For example, the clone for an item can be manually disabled, such as by a field reliability engineering operator via a tool, regardless of whether the item is in the initial state, the clone_pending state, or the clone_ready state. The clone can be disabled to repair hardware, resolve errors or faults, periodic maintenance, and the like.

[0054] Figure 5 An example of how to identify the item state and the data stored in the regional database is illustrated. When the global database is updated, the update can be replicated to the regional database in chunks. For example, Figure 5 The regional database table 500 is illustrated, where the data chunks are rows. Each row has a key 510 that identifies the corresponding data 550. As the item is updated, one or more rows can be added, deleted, or modified. In addition, a row can be added to identify the state of the item. For example, as illustrated, the key can include an identifier for the item. The row can further identify the item state and a timestamp of the time at which the item transitioned to the item state. Each time the item state changes, the row can be updated to reflect the latest item state and a timestamp indicating the time at which the transition to the state occurred.

[0055] Table 500 can include data for one or more items. In the case of multiple items, these items can be in different states at different times. Thus, each item will have a separate identifier, a separate item state, and a separate timestamp. For example, while item A can have entered the Clone Pending state at time tl, item B can have entered the Clone Ready state at time t2. As each of item A and item B transitions to different states, their corresponding rows can be updated individually to reflect the latest state and timestamp.

[0056] Although the table is described above as being located in the global clone, according to other examples it can be stored elsewhere in the region, such as in the region database.

[0057] Figure 6 A timing diagram is illustrated in which the states of various items are recorded in the region database. The timing of updates to the global database and global clone for an item is the same as for Figure 3 . However, since reads are handled based on item state, the way read requests are handled is different than explained above in connection with Figure 3 .

[0058] Version 1 is updated in the global database before time 1, but is not replicated in the global clone until after time 2. At time T a , the item enters the Invalid state when it is first enabled in the global clone. Thus, if a customer request is processed at time 1, the business logic will see version 1 from the global database.

[0059] After time T a and time 2, version 1 is enabled in the global clone. However, at this point, version 2 is already available in the global database. At time 2, the business logic will still see version 1 from the global database, since it will ignore the global clone when the item is in the Invalid state. Similarly, at time 3, when the item is still in the Invalid state, the business logic will ignore the global clone and read version 2 from the global database.

[0060] At time T b , the item is detected in the global clone. Thus, the item transitions from the Invalid state to the Clone Pending state. The timestamp T b is recorded as the time when the item entered the Clone Pending state. Thus, a customer request processed at time 4, while the item is still in the Clone Pending state, will reference the global database. However, the request is not referencing the global database at time 4, i.e., after version 3 has been implemented in the global database, but at time T b. , i.e., when the item entered the Clone Pending state. Thus, the read request will yield version 2 from the global database.

[0061] At time Tc, the item enters the clone_ready state. For example, the control plane storage can have verified that the global clone is up to date compared to the global database as of time Tb. Thus, the item transitions to the clone_ready state at time Tc, and a timestamp Tc is recorded.

[0062] A read processed at time 5 will yield version 2 of the feature from the global clone. By time 6, version 3 of the item has been implemented in the global clone. Thus, a read at time 6 will yield version 3 from the global clone.

[0063] In addition to the operations described above and illustrated in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the exact order described below. Rather, various steps can be handled in different order or simultaneously, and steps can also be added or omitted.

[0064] Figure 7 is a flowchart illustrating an example method of setting an item state. For example, the item can be a customer, a catalog, an application, or any other replication unit.

[0065] In block 710, a global clone in a region is enabled for a particular item in a global database. For example, the item can be sent from the global database to the global clone in the region for replication. The item can be sent in data blocks over a network. There can be a large physical distance between the global database and the global clone. Thus, there can be a delay between the time the global database starts sending data, the time the global clone is first enabled, and the time the item is fully implemented in the global clone.

[0066] In block 720, the state of the item is set to a clone_pending state, which indicates that the item has been enabled in the global clone but has not caught up to the global database. The item state can be updated by a data tier, such as a control plane service, such as a regional control plane storage. The item state can be indicated in a regional database along with the data of the item. For example, a row can be appended to a database table of the item, where the row indicates the item state.

[0067] In block 730, a timestamp is recorded for the time the item state is updated. For example, if the item state is set to the clone_pending state at time t, then the time t is recorded. The timestamp can be recorded along with an indication of the item state, such as in a regional database.

[0068] In block 740, transactions for the item in the clone_pending state are observed. For example, a maintenance activity in the control plane storage can detect whether the item has any transactions, such as read requests.

[0069] In block 750, it is determined whether the global clone of the item has caught up to the global database. For example, it can be determined whether the global clone has fully implemented the version of the item that was implemented in the global database at the recorded timestamp. This determination can be made by the maintenance activity stored by the control plane. If the global clone has not caught up, the item can remain in the clone-pending state and the maintenance activity can wait. For example, the maintenance activity can continue to watch the transaction until it determines that the global clone has caught up to the recorded timestamp.

[0070] If it is determined in block 750 that the global clone of the item has caught up to the recorded timestamp, the item is transitioned to the clone-ready state in block 760, which indicates that the global clone of the item has caught up. For example, the indication of the item state in the database can be updated to reflect the new state. In addition, a new timestamp corresponding to the time of the transition from the clone-pending state to the clone-ready state is recorded along with the updated state.

[0071] Figure 8 is a flowchart illustrating an example method of servicing read requests based on item states. The item state is set in block 820, and a timestamp of when the item state was set is recorded in block 830. For example, as discussed above in connection with blocks 720 and 730 of Figure 7 the maintenance activity stored by the control plane can set the item state and record it in the regional database along with a timestamp.

[0072] In block 840, a request to read the item data is received, for example, at the regional stack. The request will be handled according to the state of the item.

[0073] In block 850, it is determined whether the global clone of the item is in the invalid state. If so, the read request will be serviced by the global database (block 855).

[0074] In block 860, it is determined whether the global clone of the item is in the pending state. If so, the global clone is not ready to service the read request. Accordingly, the read request is redirected to the global database at the recorded timestamp (block 865). For example, the data in the global database can be preserved for a period of time, such as hours, days, etc., even though an updated version of the item that later changed the data can have been implemented. The read request is directed to the global database at a time corresponding to the time the timestamp was recorded, even though the request can have been received at a later time and the global database can have been further updated.

[0075] In block 870, it is determined whether the global clone is in the ready state, in which it has been updated with the data in the global database at the recorded timestamp. If the global clone is in the ready state, the read request can be serviced by reading the data from the clone database (block 875).

[0076] If the clone is not in a pending or ready state, it can be determined that the clone is disabled. Thus, in block 880, the read request is ignored.

[0077] The above-described systems and methods have the advantage that they provide consistent reads in global clones that are replicated from a global database. By redirecting only some requests to the global database, and by having the global clone serve other requests, the load on the global database is reduced. Further, requests served by the global clone can produce data faster than the global database, such as if the request is issued from the same region as the global clone. In addition, since requests redirected to the global database are redirected to a corresponding earlier point in time when the global clone state changed, data read from the global database will be consistent with the global clone’s data, reducing unexpected effects.

[0078] Unless otherwise indicated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. Additionally, the provision of the examples described herein, as well as clauses phrased in the present tense, should not necessarily be interpreted to mean that an aspect of the disclosure has to occur in every instance of the disclosure. Moreover, the word “example” is used an exemplary nature, to indicate that one has been given as a non-limiting example. Additionally, the terms “comprise,” “comprises,” “comprising,” “include,” “includes,” and “including” should be construed to be inclusive (i.e., to “comprise,” “include,” and “including” one or more elements), unless otherwise indicated as otherwise exclusively restricted or limited by context. Furthermore, the word “or” as used herein means any one member of a set or all of the members of a set.

Claims

1. A method for dispatching read requests, comprising: Automatically invalidate the status of new projects, where all requests to read data for said project generate data from the global database, wherein said project has not yet been enabled in the global clone; Enable the global clone for the project in the first region, wherein enabling the global clone includes receiving data for the project from the global database at the global clone for replication; When the new project is detected, one or more processors in the first region change the status of the project from the invalid state to the pending state, and the one or more processors in the first region record a timestamp corresponding to the time t1 when the status was changed from the invalid state to the pending state, wherein... If the project is in the pending state, a request to read data for the project received by the one or more processors in the first region is redirected to the global database at time t1+x, but implemented by the global database version at time t1 corresponding to the timestamp; The one or more processors in the first region determine whether the global clone has been updated with the same data in the global database up to the timestamp t1, and If the global clone has been updated using the same data, the state of the project is changed to a ready state and a timestamp corresponding to the time when the state was set to the ready state is recorded, wherein if the state of the project is in the ready state, the data from the global clone is used to implement a request for reading data for the project received by the one or more processors in the first region.

2. The method of claim 1, further comprising, when the project is in the pending state, having the one or more processors in the first region observe the transactions of the project; wherein, Determining whether the global clone has been updated is in response to a transaction that observes the project.

3. The method according to claim 2, wherein, The transaction includes writing the transaction to the project's global database.

4. The method according to claim 1, further comprising: Change the status of the project from the ready state back to the pending state; as well as A second timestamp is used to record the time when the state changes from the ready state to the pending state.

5. The method according to claim 1, wherein, Recording the timestamp includes appending the timestamp to a row in a data table in the global clone or regional database.

6. The method of claim 5, further comprising recording the timestamp together with an indication of the state and an identifier of the item.

7. A system for dispatching read requests, comprising: A global clone in the first region, configured to copy data from the global database for the project; as well as One or more computing devices, the one or more computing devices being configured to: Automatically invalidate the status of new projects, where all requests to read data for said project generate data from the global database, wherein said project has not yet been enabled in the global clone; Enable the global clone for the project in the first region, wherein enabling the global clone includes receiving data for the project from the global database at the global clone for replication; When the new project is detected, the project's status is changed from the invalid state to the pending state, and a timestamp corresponding to the time t1 when the status was changed from the invalid state to the pending state is recorded, wherein, If the project is in the pending state, the request to read data for the project is redirected to the global database at time t1+x, but implemented by the global database version at time t1 corresponding to the timestamp; Determine whether the global clone has been updated with the same data in the global database up to the timestamp t1, and If the global clone has been updated using the same data, the state of the project is changed to a ready state and a timestamp corresponding to the time when the state was set to the ready state is recorded, wherein if the state of the project is in the ready state, the data from the global clone is used to implement a request to read data for the project.

8. The system according to claim 7, wherein, The one or more computing devices are further configured to observe transactions of the project while the project is in the pending state; wherein determining whether the global clone has been updated is in response to the observation.

9. The system according to claim 8, wherein, The transaction includes writing the transaction to the project's global database.

10. The system according to claim 7, wherein, The one or more computing devices are further configured to: Change the status of the project from the ready state back to the pending state; and A second timestamp is used to record the time when the state changes from the ready state to the pending state.

11. The system according to claim 7, wherein, Recording the timestamp includes appending the timestamp to a row in a data table in the global clone or regional database.

12. The system according to claim 11, wherein, The one or more computing devices are also configured to record the timestamp together with an indication of the state and an identifier of the item.

13. A non-transitory computer-readable medium storing instructions executable by one or more processors to perform a method of dispatching a read request to a global clone, the method comprising: Automatically invalidate the status of new projects, where all requests to read data for said project generate data from the global database, wherein said project has not yet been enabled in the global clone; Enable the global clone for the project in the first region, wherein enabling the global clone includes receiving data for the project from the global database at the global clone for replication; When the new project is detected, one or more processors in the first region change the status of the project from the invalid state to the pending state, and the one or more processors in the first region record a timestamp corresponding to the time t1 when the status was changed from the invalid state to the pending state, wherein... If the project is in the pending state, a request to read data for the project received by the one or more processors in the first region is redirected to the global database at time t1+x, but implemented by the global database version at time t1 corresponding to the timestamp; The one or more processors in the first region determine whether the global clone has been updated with the same data in the global database up to the timestamp t1, and If the global clone has been updated using the same data, the state of the project is changed to a ready state and a timestamp corresponding to the time when the state was set to the ready state is recorded, wherein if the state of the project is in the ready state, the data from the global clone is used to implement a request for reading data for the project received by the one or more processors in the first region.

14. The computer-readable medium according to claim 13, wherein, The execution of the instruction also provides the ability to observe the transactions of the project while the project is in the pending state; wherein determining whether the global clone has been updated is in response to observing the transactions of the project.

Images