Switching to eventually consistent database replication

Abstract

To provide a method of and a system for consistent reading from a global database duplicated to one or a plurality of global clones.SOLUTION: The present invention is directed to a method mounted with a state machine and carried out independently in each project of each region. Based on a state of project obtained when a read request is received, the read request is processed by a global database, or processed by a global clone, or is disregarded as an invalid request.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 16 / 682,513, filed November 13, 2019, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] background Infrastructure-as-a-Service is a service that It is supported by a global network of computing devices that act as role plane and data plane. Data stores provide data for such large control plane services. There are multiple separate regions within the global network. Some computing resources are isolated within these regions, while others are global. However, the control plane typically does not communicate directly with the global database. A global stack is maintained for the global resources. A stack can be, for example, a set of computing resources with business logic. For example, if a customer wants to create a virtual machine, a request is sent to the region stack, and the business logic implemented in the region stack works with the global database to get the requested information.

[0003] One issue with the above setup is that all the regions are interconnected by network links, and the links may experience outages. For this reason, the global database is replicated to the global clone database. However, because the global database and the global clones are updated asynchronously, there is typically a delay between updating the global database and replicating such updates to the global clone database. For example, the delay may range from tens of milliseconds to tens of seconds. Thus, reading from the global database and the global clone database at a given time may result in inconsistent reads. This may cause multiple issues and side effects in business logic. As an example, when a control plane store reads data at a first timestamp and then at a second consecutive timestamp, the data from the first timestamp is always older than, or possibly identical to, the data from the second timestamp. This immutable relationship ensures integrity. For example, a virtual machine resource that has been "deleted" cannot be "undeleted." However, reads from the global database and subsequent successive reads from the global clones do not traditionally provide the same immutable relationship. Rather, it may be possible to read a virtual machine that has been "removed" from the global database, and then a short time later, when the clone is read, it may be found to still be "active." Summary of the Invention [Means for solving the problem]

[0004] Quick Overview The present disclosure provides consistent reads in a global database that is replicated to one or more global clones. For each project in each region, a state machine is implemented and executed independently. A project can be, for example, a grouping of resources that belong to the same customer. Based on the state of the project when the read request is received, the read request is either processed by the global database or by the global clone. It will either be processed by the loan or ignored as an invalid request.

[0005] One aspect of the disclosure provides a method, the method including enabling a global clone in a first region for a project, the enabling of the global clone including receiving data of the project from a global database in the global clone for replication. The method further includes one or more processors in the first region setting a state of the project to a pending state, the one or more processors in the first region recording a timestamp corresponding to a time when the state was set to the pending state, the one or more processors in the first region receiving a request to read data of the project, and redirecting the request to the global database at a time corresponding to the timestamp when the state of the project is in the pending state. According to some examples, the method may further include the one or more processors in the first region determining whether the global clone has been updated with data identical to data in the global database as of the timestamp, and if the global clone has been updated with the same data, changing the state of the project to a ready state.

[0006] The method may further include the one or more processors in the first region receiving a second request to read data from the project and satisfying the request with data from the global clone when the project is in the ready state. In another example, the method may further include changing the state of the project from the ready state back to the pending state and recording a second timestamp for the time of change from the ready state to the pending state.

[0007] Another aspect of the present disclosure provides a system including a global clone in a first region, the global clone configured to replicate data from a global database of projects, the system further including one or more computing devices configured to receive data of the projects from the global database in the global clone for replication, set a state of the projects to a pending state, record a timestamp corresponding to a time when the state was set to the pending state, receive a request to read data of the projects, and redirect the request to the global database at a time corresponding to the timestamp when the state of the projects is in the pending state.

[0008] In some examples, the one or more computing devices may be further configured to determine whether the global clone has been updated with data that is identical to data in the global database as of the timestamp, and, if the global clone has been updated with the identical data, change the state of the project to a ready state.

[0009] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium storing instructions executable by one or more processors to perform a method for processing a read request to a global clone, the method including receiving data of a project from a global database at the global clone for replication, setting a state of the project to a pending state, recording a timestamp corresponding to a time when the state was set to the pending state, and and redirecting the request to the global database at a time corresponding to the timestamp when the state of the project is in the pending state. According to some examples, execution of the instructions may further provide for determining whether the global clone has been updated with data identical to data in the global database as of the timestamp, and changing the state of the project to a ready state if the global clone has been updated with the same data. Execution of the instructions may further provide for receiving a second request to read data from the project, and satisfying the request with data from the global clone when the project is in the ready state. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a system according to an aspect of the present disclosure. [Diagram 2] FIG. 2 is a block diagram showing an example of details of the components of the system of FIG. 1. [Diagram 3] FIG. 11 is a timing diagram illustrating an example of a delay between a global database and a global clone according to an aspect of the present disclosure. [Figure 4] FIG. 11 is a state diagram illustrating examples of different states of a project in a global clone according to aspects of the present disclosure. [Diagram 5] FIG. 2 is a diagram illustrating an example of a data chunk according to an aspect of the present disclosure. [Figure 6] FIG. 11 is another timing diagram illustrating example transactions according to aspects of the present disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating an example of a method for setting a project status according to an aspect of the present disclosure. [Figure 8] FIG. 11 is a flow diagram illustrating an example method for processing a read request based on a project state according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description 1 illustrates an example system 100 that includes a global database 110 and multiple regional databases 120, 160. Within each regional 105, 155 are zone databases 140, 150, 180, 190. For example, the zone in which the zone database 140, 150 resides may be a subarea of ​​the region served by regional database 120. Similarly, the zone in which the zone database 180, 190 resides may be a subarea of ​​the region served by regional database 160. The zones may have high bandwidth, low latency network connections to other zones in the same region. Although only a few regional 105, 155 are illustrated, it should be understood that a global system may include any number of regional 105, 155, and that each regional 105, 155 may have any number of zones.

[0012] Each region 105, 155 also includes a global clone database 130, 170. The global database 110 is asynchronously replicated to the global clone database 130, 170. In some examples, all data from the global database 110 may be replicated to both clones 130, 170. According to other examples, some data from the global database 110 is replicated to a first clone 130 and other data from the global database 110 is replicated to a second clone 170. For example, a particular project may be replicated to clones in a particular region.

[0013] Each database is controlled by a stack. For example, the global stack 112 serves as the control plane for the global database 101. The regional stacks 122, 162 serve as the control plane for the regional databases 120, 170, respectively. Within the first region 105, the zone stacks 142, 152 , serve as the control plane for zone databases 140, 150, respectively. Within the second domain 155, zone stacks 182, 192 serve as the control plane for zone databases 180, 190, respectively.

[0014] The control plane store provides a data layer for the control plane services. Each database and stack may have a dedicated control plane store. For example, as shown in FIG. 1, the global control plane store 114 provides a data layer for the global stack 112. The regional control plane stores 124, 164 provide a data layer for the regional stacks 122, 162, respectively. The zonal control plane stores 144, 154, 184, 194 provide a data layer for the zonal stacks 142, 152, 182, 192, respectively.

[0015] The global database 110 can retrieve past versions of data. For example, when an update to the database 110 occurs, rather than immediately erasing the previous information, the previous information may be retained for a period of time. This period may be minutes, hours, days, etc. Thus, the global database 110 allows the control plane store 114 service to retrieve data as of any timestamp in the past (e.g., -30 minutes, -4 hours from the "current" timestamp).

[0016] 2 provides further details regarding the stack and control plane store. A domain stack 202 includes a set of computing resources. For example, stack 202 includes one or more processors 240. Processor 240 is configured to execute data and instructions in memory 230. For example, processor 240 may execute business logic 235 to control a project's virtual infrastructure (IaaS).

[0017] The processor 240 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a specialized component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, the stack 202 may include specialized hardware components for performing specific computing operations.

[0018] The stack 202 may use the database 200 to persist the state of resources processed by the processor 240. teeth , may be any type of non-transitory computer-readable medium capable of storing information accessible by the processor 240 (such as a hard drive, solid-state drive, tape drive, optical storage, memory cards, ROM, RAM, DVD, CD-ROM, writable and read-only memory, etc.).

[0019] The memory 230 may store information accessible by the processor 240 , including instructions that may be executed by the processor 240 and that may be retrieved, manipulated or stored by the processor 240 .

[0020] The instructions may be a set of instructions that are executed directly by the processor 240 (such as machine code) or indirectly by the processor 240 (such as a script). In this regard, the terms "instructions," "steps," and "program" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor 240, or in other types of computer languages ​​(including scripts or collections of independent source code modules that are interpreted on demand or precompiled). The functions, methods, and routines of the instructions are , as described in further detail above and in the method examples below.

[0021] The data may be retrieved, stored or modified by the processor 240 in accordance with the instructions, and may be formatted in a computer readable form, such as, but not limited to, binary values, ASCII or Unicode. Additionally, the data may include sufficient information to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the relevant data.

[0022] Memory 230 includes business logic 235 for controlling data in database 200. For example, business logic 235 may determine how transactions with database 200 are handled. Depending on the state of the project's global clone, business logic 235 processes read requests differently. For example, as described further herein, if the project's global clone is pending, business logic 235 reads data from the global database at the time corresponding to the timestamp at which the clone became pending. Once the project's global clone is ready, business logic 235 processes the request by reading the data from the global clone.

[0023] 2 functionally illustrates the processor, memory, and other elements of computing device 200 as being within the same block, a processor, computer, computing device, or memory may actually include multiple processors, computers, computing devices, or memories that may or may not be housed within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a housing different from that of computing device 200. Thus, reference to a processor, computer, computing device, or memory will be understood to include reference to a collection of processors, computers, computing devices, or memories that may or may not operate in parallel. For example, computing device 200 may include server computing devices that operate as a load-balanced server farm, distributed system, and the like. Additionally, while some functions described below are illustrated as occurring on one computing device having one processor, various aspects of the subject matter described herein may be implemented by multiple computing devices that communicate information, for example, over a network.

[0024] The regional control plane store 204 may include one or more processors 270, memory 260, and other components for providing a data layer for control plane services. The control plane store 204 may further include one or more state machines 280 for projects replicated to global clones in the region. The state machines 280 may be, for example, computing devices programmed to change from one state to another in response to certain inputs or conditions. As further described herein in connection with the example of FIG. 4, the states may include a disabled state, a pending state, and a ready state. In some examples, the states may further include a disabled state.

[0025] The memory 260 may store information accessible by the processor 270, including instructions 262 that may be executed by the processor 270. The memory may also include data 264 that may be retrieved, manipulated, or stored by the processor 270. The memory 260 may be any type of non-transitory computer-readable medium capable of storing information accessible by the processor 270, such as a hard drive, solid-state drive, The processor 270 may be a known processor or other lesser known type of processor. Alternatively, the processor 270 may be a dedicated controller (such as an ASIC).

[0026] The instructions 262 may be a set of instructions that are executed directly by the processor 270 (such as machine code) or indirectly by the processor 270 (such as a script). In this regard, the terms "instructions," "steps," and "program" may be used interchangeably herein. The instructions 262 may be stored in object code format for direct processing by the processor 270, or in other types of computer languages ​​(including a script or a collection of independent source code modules that are interpreted on demand or precompiled).

[0027] The data 264 may be retrieved, stored or modified by the processor 270 according to the instructions 262. For example, the data 264 may be stored in a computer register or in a relational database as a table having multiple different fields and records or an XML document, although the systems and methods are not limited by any particular data structure. The data 264 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII or Unicode. Furthermore, the data 264 may include sufficient information to identify related information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0028] Instructions 262 may include performance maintenance activity 267. The maintenance activity may be executed to determine a condition of the project in the global clone and update a state of the project's global clone based on the determined condition. For example, maintenance activity 267 may determine that the project's global clone has been enabled and update the project's global clone from a disabled state to a pending state. Further, maintenance activity 267 may determine that the project's global clone has caught up to a particular point in time in the global database. Thus, maintenance activity 267 may update the state of state machine 280 for the project to transition the project state from a pending state to a ready state.

[0029] 2 functionally depicts processor 270 and memory 260 as being in the same block, processor 270 and memory 260 may actually include multiple processors and memories that may or may not be stored in the same physical housing. For example, some of instructions 262 and data 264 may be stored on a removable CD-ROM, while other parts may be stored in a read-only computer chip. Some or all of the instructions and data may be stored physically remote from processor 270, but still accessible by processor 270. Similarly, processor 270 may actually include a collection of processors that may or may not operate in parallel.

[0030] Database 200 may be a storage device or multiple storage devices (hard drives, random access memory, disks, disk arrays, tape drives, etc.). Database 200 may implement any of several architectures and technologies, including direct attached storage (DAS), network attached storage (NAS), storage The database 200 may be configured to access, store, or use any of a variety of storage devices, including, but not limited to, a storage area network (SAN), a fiber channel (FC), a fiber channel over Ethernet (FCoE), or a mixed architecture network. Additionally, in some examples, the database 200 may include a virtualized or containerized environment. For example, the database 200 may include one or more virtual machines running on a host machine. The database 200 may store, for example, data files, documents, code, schemas, persistence frameworks, applications, or any of a variety of other information or tools typically stored in a database.

[0031] The stack 202 may process requests from client devices (not shown), such as over a network. A client device may be configured with a processor, memory, instructions, and data. A client may be a personal computer intended for use by a person, having all the internal components typically found in a personal computer, such as a central processing unit (CPU), a hard drive, and input / output devices, and all the components used to connect these elements to each other. A client may be any device capable of processing instructions and transmitting data between humans and other computers, including general purpose computers, tablets, mobile phones, smart watches, home assistants, video game consoles, network computers with no local storage capabilities, set-top boxes for televisions, and other network-connected devices.

[0032] A client may include an application interface module that is used to access services made available by database 200. For example, the application interface module may include subroutines, data structures, object classes, and other types of software components that are used to allow a server and a client to communicate with each other. In one aspect, the application interface module may be a software module operable in conjunction with several types of operating systems known in the art. For example, a client may be connected to a Structured Query Language (SQL) database server that may operate in conjunction with the application interface module to store and retrieve information data.

[0033] The networks and intervening nodes may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using one or more corporate proprietary communications protocols, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n or other such standards), and HTTP, as well as various combinations of the above. Such communications may be facilitated by devices capable of transmitting data to and from other computers, such as modems (e.g., dial-up, cable or fiber optic) and wireless interfaces.

[0034] FIG. 3 is a timing diagram illustrating an example of a delay between the global database and the global clone. As shown, version 1 is updated in the global database before time 1. The version can be, for example, a version of a feature (such as a project or application). Because of the delay, version 1 is not replicated to the global clone until after time 2. In this example, when version 1 is replicated to the global clone, version 2 is available in the global database. Thus, when a customer request is processed at time 1, the business logic sees version 1 from the global database. At time 2, the business The logic will still look at version 1 from the global database.

[0035] After time 3, the feature is sent and enabled in the particular geographic region served by the regional database (such as first region 105 in FIG. 1 ). Thus, a customer request processed at time 3 will generate the latest version (version 2) from the global database, since the feature has not yet been enabled in the region at time 3.

[0036] Before time 4, version 3 is implemented in the global database. However, at this time, version 2 has not yet been implemented in the global clone. Therefore, a request processed at time 4 will generate version 1 from the global clone. Although version 3 is already implemented in the global database at this point, the global clone generates data from a previous version. This can have unintended side effects. For example, if version 3 was a "delete" of a virtual network, generating version 1 from the global clone would essentially perform an "undelete" of the virtual network.

[0037] By time 5, the global clone has generated version 2 of the feature, which is still not up-to-date. It is not until time 6 that the global clone catches up with the global database. Thus, any request before time 6 may receive an inaccurate response.

[0038] Unintended effects due to differences between the global database and the global clone at different times can be resolved by implementing a state machine for each project in each domain. For example, referring back to FIG. 1, a state machine can be implemented in the regional control plane stores 124, 164 of each domain 105, 155.

[0039] 4 is a state diagram illustrating an example of different states of a project in a global clone. According to this example, the different states include an invalid state 410, a clone pending state 420, a clone ready state 430, and a disabled state 440.

[0040] A new project may be automatically placed in a disabled state 410. In this state, the project may not yet be enabled in the global clone, so any requests to read data that are processed while the project is in the disabled state will generate data from the global database.

[0041] The maintenance activity may discover the new project, for example, by scanning. For example, the maintenance activity may read all project resources in the global database and detect whether the project is new. A project may be determined to be new if the project state does not exist in a particular domain clone. If a new project is detected, the project may be set to a clone pending state 420. A timestamp is recorded of the time the project changed from an invalid state 410 to a clone pending state 420. For example, the timestamp may be recorded in the domain database along with an indication of the project's state, as further described below in connection with FIG. 5.

[0042] All global data requests to the regional control plane store for a project in the clone pending state 420 are redirected to the global database at the time corresponding to the recorded timestamp. For example, a project may transition from the invalid state 410 to the clone pending state 420 at time t1, and time t1 is recorded. A read request may be received at a later time t1+x, but if the project is still in the clone pending state 420, the read request is redirected to the global database at time t1. The global database may have been updated with a newer version of the project between time t1 and time t1+x, but the version at time t1 is still generated for consistency.

[0043] Also, a maintenance activity in the region control plane store may detect a transaction for a project in the clone pending state 420. For example, the maintenance activity may detect that a threshold number of reads or writes for the project have occurred. The threshold number may be equal to or greater than one. When this occurs, the maintenance activity may determine whether to transition the project to the clone ready state 430. For example, the maintenance activity may check whether the global clone in the region has caught up with the global database for the project. If the latest version available in the global clone is identical to the version available in the global database at the time of the recorded timestamp, the global clone may be considered to have caught up. If the global clone has caught up, the maintenance activity changes the state of the project to the clone ready state 430. In the clone ready state 430, all reads are satisfied by the global clone.

[0044] The maintenance activity may further detect when a project in the clone ready state 430 becomes inconsistent with the global database. For example, the maintenance activity may periodically check whether the global clone is still up to date with the global database. In another example, the maintenance activity may detect updates to the project's global database and, accordingly, check whether the updates are replicated to the global clone. If it is determined that the project's global clone is inconsistent with the global database, the state of the project may transition from the clone ready state 430 back to the clone pending state.

[0045] A project can be transitioned to the disabled state 440 at any time. For example, a project's clones can be manually disabled, such as by a site reliability engineering operator via a tool, whether the project is in an initial state, a clone pending state, or a clone ready state. Clones can be disabled for hardware repairs, error or fault resolution, routine maintenance, and the like.

[0046] FIG. 5 shows an example of how a project state may be identified along with data stored in a domain database. When the global database is updated, the updates may be replicated to the domain database in chunks. For example, FIG. 5 shows a domain database table 500 where the chunks of data are rows. Each row has a key 510 that identifies the corresponding data 550. When a project is updated, one or more rows may be added, deleted, or modified. In addition, a row may be added that identifies the state of the project. For example, as shown, the key may include an identifier for the project. The row may further identify the project state and a timestamp of when the project transitioned to that project state. Each time the project state changes, the row may be updated to reflect the latest project state and a timestamp indicating when the transition to that state occurred.

[0047] Table 500 may contain data for one or more projects. When multiple projects are included, the projects may be in different states at different times. Thus, , each project has a separate identifier, a separate project state, and a separate timestamp. For example, project A may have entered a clone pending state at time t1, while project B may have entered a clone ready state at time t2. As projects A and B each transition to a different state, their corresponding rows may be updated individually to reflect the latest state and timestamp.

[0048] Although the tables are described above as being in the global clone, according to other examples they may be stored elsewhere in the domain (such as in the domain database).

[0049] Figure 6 shows a timing diagram in which the state of individual projects is recorded in the domain database. The timing of the project global database updates and global clone updates is identical to that of Figure 3. However, read requests are handled differently than described above in connection with Figure 3, since reads are processed based on project state.

[0050] Version 1 is updated in the global database before time 1, but is replicated to the global clone after time 2. a In, when a project is first enabled in a global clone, the project is in a disabled state, so when a customer request is processed at time 1, the business logic will see version 1 from the global database.

[0051] Time T a And after time 2, version 1 is enabled in the global clone, but this time version 2 is available in the global database. At time 2, the business logic will still see version 1 from the global database because it ignores the global clone when the project is in a disabled state. Similarly, at time 3, when the project is still in a disabled state, the business logic will ignore the global clone and read version 2 from the global database.

[0052] Time T b At timestamp T, the project is found in the global clone. Therefore, the project transitions from the disabled state to the clone pending state. bis recorded as the time when the project entered the clone pending state. Thus, a customer request that is processed at time 4 while the project is still in the clone pending state will reference the global database. However, the request will not reference the global database at time 4, when version 3 is already implemented in the global database, but rather the global database at time T b Therefore, a read request will generate version 2 from the global database.

[0053] At time Tc, the project is in a clone-ready state. For example, the control plane store may have determined that the global clone is up to date compared to the global database as of time Tb. Thus, the project transitions to a clone-ready state at time Tc and a timestamp Tc is recorded.

[0054] A read processed at time 5 produces version 2 of the feature from the global clone. By time 6, version 3 of the project has been implemented in the global clone. Thus, a read at time 6 produces version 3 from the global clone.

[0055] In addition to the operations described above and illustrated in the figures, various operations are described below. It should be understood that the operations below do not have to be performed in the exact order described below. Rather, various steps may be processed in a different order or simultaneously, and steps may be added or omitted.

[0056] 7 is a flow diagram illustrating an example of a method for setting the state of a project. A project may be, for example, a customer, a directory, an application, or other unit of replication.

[0057] At block 710, a global clone in a region is enabled for a particular project in the global database. For example, the project may be sent from the global database to the global clone in the region for replication. The project may be sent in data chunks over a network. There may be significant physical distance between the global database and the global clone. Thus, there may be a delay between when the global clone is first enabled and the global database starts sending data and when the project is fully implemented in the global clone.

[0058] At block 720, the state of the project is set to clone pending to indicate that the project has been enabled in the global clone but has not yet caught up in the global database. The project state may be updated by a data layer for a control plane service, such as a regional control plane store. The project state may be indicated in a regional database along with the project's data. For example, a row may be added to a database table for projects indicating the project state.

[0059] At block 730, a timestamp of the time the project state was updated is recorded. For example, if the project state was set to clone pending at time t, then time t is recorded. The timestamp may be recorded in a domain database or the like along with an indication of the project state.

[0060] At block 740, transactions for the project in the clone pending state are observed. For example, maintenance activity in the control plane store may detect whether there have been any transactions (such as read requests) for the project.

[0061] At block 750, it is determined whether the global clone of the project has caught up with the global database. For example, it may be determined whether the global clone has fully implemented the version of the project that was implemented in the global database at the recorded timestamp. The determination may be made by a maintenance activity of the control plane store. If the global clone has not caught up, the project may remain in a clone pending state and the maintenance activity may wait. For example, the maintenance activity may continue to observe transactions until it determines that the global clone has caught up with the recorded timestamp.

[0062] If, at block 750, it is determined that the global clone of the project has caught up to the recorded timestamp, then, at block 760, the project is transitioned to a clone-ready state to indicate that the global clone of the project has caught up. For example, the indication of the project state in the database may be updated to reflect the new state. Additionally, along with the updated state, the indication may be transitioned from a clone pending state to a clone ready state. A new timestamp is recorded corresponding to the time of transition to the ready-to-run state.

[0063] 8 is a flow diagram illustrating an example of a method for processing a read request based on a project state, setting the project state at block 820 and recording a timestamp when the project state was set at block 830. For example, as described above in connection with blocks 720 and 730 of FIG. 7, the project state may be set by a maintenance activity in the control plane store and recorded in the domain database along with a timestamp.

[0064] At block 840, a request to read data for a project, for example in a region stack, is received. The request is handled according to the state of the project.

[0065] At block 850, it is determined whether the global clone of the project is in an invalid state. If so, the read request is processed by the global database (block 855).

[0066] At block 860, it is determined whether the global clone for the project is in a pending state. If so, the global clone is not ready to process the read request. Thus, the read request is redirected to the global database at the time of the timestamp (block 865). For example, data in the global database is stored for a period of time (hours, days, etc.) even though the data may have changed later due to an updated version of the project being implemented. The read request is directed to the time in the global database that corresponds to the time of the recorded timestamp, although the request may have been received since then and the global database may have been further updated since then.

[0067] At block 870, it is determined whether the global clone is in a ready state, having been updated with the data that was in the global database at the recorded timestamp. If the global clone is in a ready state, the read request may be processed by reading the data from the clone database (block 875).

[0068] If the clone is not in a pending or ready state, then the clone may be determined to be disabled, and therefore the read request is ignored at block 880.

[0069] The above system and method are advantageous in that it provides consistent reads in the global clones replicated from the global database. By redirecting only some requests to the global database and processing others by the global clone, the load on the global database is reduced. Furthermore, requests processed by the global clone may generate data faster than the global database, such as when the requests are issued from the same region as the global clone. Furthermore, because requests redirected to the global database are redirected to an earlier point in time corresponding to when the state of the global clone changed, the data read from the global database is consistent with the data of the global clone, reducing unintended effects.

[0070] Unless otherwise specified, the above-described alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. These and other variations and combinations of the above features may be utilized without departing from the subject matter defined by the claims, and therefore the above description of the embodiments should not be taken as limiting the subject matter defined by the claims. The examples described herein should not be construed as limiting the subject matter of the claims to specific examples, but rather as illustrative. Additionally, the provision of examples herein, and clauses using phrases such as "such as" and "including," should not be construed as limiting the subject matter of the claims to specific examples, but rather, the examples are intended to illustrate only one of many possible embodiments. Additionally, the same reference numbers in various figures may identify the same or similar elements.

Claims

1. 1. A method for providing an Infrastructure-as-a-Service (IaaS) to a customer, the Infrastructure-as-a-Service (IaaS) being implemented on a plurality of computing devices that communicate information over a network, the method comprising: and enabling a global clone in a region for a project, the project corresponding to data belonging to a same customer in a global database, the enabling of the global clone including receiving data for the project from a global database in the global clone for replication, the method further comprising: one or more processors in the region setting the state of the project to a pending state; detecting a transaction for the project in the pending state, the transaction comprising a read transaction for data of the project, the detecting the transaction comprising receiving a request to read data of the project, the method further comprising: determining, using the one or more processors based on the detected transactions of the project, whether the global clone of the project has caught up with the global database, the determining including determining whether the global clone has been updated with data that is the same as data in the global database. transitioning the project from the pending state to a ready state upon determining that the global clone has caught up with the global database.

2. The method of claim 1, further comprising: when the state of the project is set to a pending state, the one or more processors in the region record a timestamp corresponding to the time the state was set to the pending state.

3. 3. The method of claim 2, further comprising: when the one or more processors in the region receive a request to read data for the project, if the processor determines that the state of the project is in the pending state, redirecting the request to read data for the project to the global database stored at the time corresponding to the timestamp.

4. Determining whether the global clone has caught up with the global database includes:

4. The method of claim 2 or 3, further comprising the one or more processors in the region determining whether the global clone has been updated with data that is identical to data in the global database as of the time of the timestamp.

5. The method of any one of claims 1 to 4, wherein the transaction comprises a write transaction to the global database of the project.

6. when the one or more processors in the region receive a request to read data from the project; The method of any one of claims 1 to 5, further comprising: if it is determined that the project is in the ready state, processing the request using data from the global clone.

7. The method of claim 2 , wherein recording the timestamp comprises adding the timestamp to a row of a data table in the global clone or in a regional database.

8. The method of claim 7, wherein adding the timestamp includes adding the timestamp to the row along with an indication of the state and an identifier of the project.

9. The method according to any one of claims 1 to 8, wherein detecting the transaction comprises detecting whether any request for the project has been received due to a maintenance activity in a control plane.

10. A system for providing a customer with an infrastructure-as-a-service (IaaS) implemented on a plurality of computing devices that communicate information over a network, comprising: A global clone in a first region, the global clone being configured to replicate data of projects from a global database, the projects corresponding to data belonging to the same customer in the global database, the system further comprising: one or more computing devices, receiving data for the project from the global database in the global clone for replication; Setting the state of the project to pending; detecting transactions for the project that are in the pending state, the transactions including read transactions for data of the project, the detecting the transactions including receiving a request to read data of the project; determining whether the global clone of the project has caught up with the global database based on the detected transactions, the determining including determining whether the global clone has been updated with data that is the same as data in the global database; The system is configured to transition the project from the pending state to a ready state upon determining that the global clone has caught up with the global database.

11. The one or more computing devices further include: The system of claim 10 , further configured to record a timestamp corresponding to a time when the state was set to the pending state.

12. The one or more computing devices further include: receiving a request to read data for the project; 12. The system of claim 11, further configured to, if it determines that the state of the project is in the pending state, redirect the request to read data of the project to the global database stored at the time corresponding to the timestamp.

13. In determining whether the global clone has caught up with the global database, the one or more computing devices:

13. The system of claim 11 or 12, configured to determine whether the global clone has been updated with data that is identical to data in the global database as of the time of the timestamp.

14. The system of claim 10 , wherein the transaction comprises a write transaction to the global database for the project.

15. The one or more computing devices further include: receiving a request to read data from the project; The system of any one of claims 10 to 14, further configured to process the request using data from the global clone if it determines that the project is in the ready state.

16. The system of claim 11 , wherein recording the timestamp comprises adding the timestamp to a row of a data table in the global clone or in a regional database.

17. The system of claim 16, wherein adding the timestamp includes adding the timestamp to the row along with an indication of the state and an identifier of the project.

18. A program for causing one or more processors to carry out the method according to any one of claims 1 to 9.

Images