Techniques for safe database migration with no downtime

Abstract

Techniques are disclosed that enable efficient data migration between data stores without downtime. A distributed computing system can be implemented using an initial data store and a target data store. During the migration of a portion of data from the initial data store to the target data store, the distributed computing system can receive a request to create a data entity or to launch a workflow instance in a data store. The system can determine whether the initial data store or the target data store is designated as a primary data store for handling the request. The system can also determine whether a key associated with the request exists in the initial data store or the target data store. If the key exists in either of the data stores, the system can abort the creation of the data entity.

Description

[Background technology]

[0001] background Cloud-based services provide solutions for handling tasks that may result in storing or updating data in a data store or database via transactions. In a multi-tenant distributed computing system, customer data may be migrated from one data store to another when the requirements of the data storage backend change. Traditionally, customer data migration incurs downtime costs that represent loss of functionality of customer processes while the computing environment is migrated from the initial data store to the target data store. Thus, there is a need for improved techniques for migrating data from one data store to another while maintaining functionality of customer processes.

[0002] Technical Field TECHNICAL FIELD This disclosure relates to distributed computing systems. More particularly, this disclosure describes techniques for migrating databases within a distributed computing system without incurring database downtime. Summary of the Invention

[0003] Short Description Embodiments of the present disclosure relate to providing a zero-downtime database migration. Migrating a database without downtime may preserve a user's ability to interact with the database before, during, and after the migration, so that users do not lose functionality while the migration is in progress. In particular, some embodiments of the present disclosure describe how to migrate some, but not all, data from one database to another.

[0004] Some embodiments provide a method for implementing an initial data store and a target data store in a distributed computing system. The data store may be a database or other data storage system. The data store may be of several different types, such as NoSQL (including key-value), relational, and multimodal. The distributed computing system may be configured to provide workflow services or other application services. For workflow-enabled embodiments, the distributed computing system may include workflow as a service (WFaaS) among a set of cloud services available to users / customers of the distributed computing system. The workflow service may handle multiple requests for transactions on the data store, both before and during data migration, so that the techniques disclosed herein may address the appropriate handling of customer requests, such as initiating or querying a workflow. A workflow may include a set of tasks or other operations that are performed to process workflow instance data.

[0005] One embodiment is directed to a method performed by a distributed computing system. The method may include implementing an initial data store and a target data store. The initial data store may have an initial store state that may be open or closed. Similarly, the target data store may have a target store state that may be open or closed. One data store is designated as a primary. The initial data store may be configured to store data associated with users (e.g., customers, tenants, etc.) of the initial data store. A portion of the data may be migrated from the initial data store to the target data store. Migrating the portion of the data may include replicating the portion of the data from the initial data store to the target data store, changing the initial store state, changing the target store state, and changing the designated primary.

[0006] While the portion of the data is being migrated, the distributed computing system may receive a customer request that involves the creation of a new data entity (e.g., a first data entity). Creating a data entity may also be referred to as launching an instance (e.g., a workflow instance), and thus the customer request may be referred to as an launch request. The creation of the first data entity may include one or more transactions with a data store. The request may include a key that may uniquely identify the request and / or other attributes. The distributed computing system may determine that the initial data store is the appropriate data store to associate with the launch request and transaction because it has been designated as the primary data store. The distributed computing system (e.g., a scheduler or other process of the distributed computing system) may also determine that the initial store state is open. If the store state is open, the target data store is checked for the presence of a key associated with the request. The key may be stored as part of the launch transaction of the first data entity. The presence of the key in the data store indicates that a data entity with an identical key has already been launched in that data store. To prevent an instance with the same key from being launched during the migration process, the distributed computing system may verify that the key does not exist in the target data store (in addition to verifying that the key does not exist in the initial data store). If the key does not exist, the first data entity may be created in the initial data store. The transaction for creating the first data entity may include committing the first data entity to the initial data store or aborting the transaction.

[0007] In another embodiment, the method may include additional operations for handling an initiation request that occurs later in the migration process. The target data store may be designated as a primary data store. The distributed computing system may receive a second request to create a second data entity associated with a second key. The distributed computing system may then determine that the target data store is designated as primary, determine the target store state, and verify that the second key does not exist in the initial data store. Based on these determinations (e.g., the target store state is open), the distributed computing system may create the second data entity in the target data store.

[0008] In some embodiments, the user data stored in the initial data store can include immutable data, temporary data, and key data. Immutable data can be data that is created in the data store and can be read but not updated or deleted. When the first and additional portions of data are migrated between the initial data store and the target data store, the portions of data can include immutable data.

[0009] In another embodiment, the method can include additional operations of handling a rollback of the data migration after a portion of the data has been replicated from the initial data store to the target data store. The cloud service (e.g., WFaaS) can create new immutable data in the target data store during the migration (e.g., the second data entity can include the new immutable data). The initial data store can be designated as a primary data store. An additional portion of the data can be transferred from the target data store to the initial data store. The additional portion of the data can include new immutable data created by the cloud service. During this transfer, the distributed computing system can receive a third request to create a third data entity. The third request can include a third key. The distributed computing system can then determine that the initial data store has been designated as a primary data store, the initial store state is open, and the third key does not exist in the target data store. The distributed computing system can then create the third data entity in the initial data store.

[0010] In some other embodiments, the method includes an act of ensuring that the initial store state is open before committing the transaction to the initial data store. The transaction may result in the creation of keys and immutable data in the data store, which data is associated with the creation of the first and second data entities.

[0011] Another embodiment is directed to a distributed computing system including one or more processors and one or more memories storing computer-executable instructions that, when executed on the one or more processors, cause the distributed computing system to implement an initial data store and a target data store. The distributed computing system may designate the initial data store as a primary data store. The instructions enable the distributed computing system to migrate a portion of the data from the initial data store to the target data store. While the data is being migrated (e.g., by replication), the distributed computing system may receive a request to create a first data entity in the initial data store. The request may include a key that may uniquely identify the request, the first data entity, a range of the first data entity, or other attributes of the request. The instructions enable the distributed computing system to determine that the initial data store has been designated as the primary data store and is therefore an appropriate data store to associate with the initiating request and transaction. The instructions may also enable the distributed computing system to determine that the initial store state is open. If the store state is open, the target data store is checked for the presence of a key associated with the request. If the key is not present, the distributed computing system stores the first data entity in the initial data store. The transaction for creating the first data entity may include committing the first data entity to an initial data store or aborting the transaction.

[0012] In yet another embodiment, the instructions can cause the distributed computing system to handle an initiation request that occurs later in the migration process. The target data store can be designated as a primary data store. The distributed computing system can receive a second request to create a second data entity associated with a second key. The instructions can cause the computing system to determine that the target data store is designated as primary, determine the target store state, and verify that the second key does not exist in the initial data store. Based on these determinations (e.g., the target store state is open), the system can create the second data entity in the target data store.

[0013] In another embodiment, the instructions can enable the distributed computing system to handle rollback of the data migration after a portion of the data has been replicated from the initial data store to the target data store. The second data entity can include new immutable data in the target data store. The initial data store can be designated as a primary data store. The instructions can cause the distributed computing system to transfer an additional portion of the data from the target data store to the initial data store. During the transfer, the distributed computing system can receive a third request to create a third data entity. The third request can include a third key. The instructions can cause the distributed computing system to determine that the initial data store has been designated as the primary data store, the initial store state is open, and that the third key does not exist in the target data store. The computing system can then create the third data entity in the initial data store.

[0014] Yet another embodiment is directed to a non-transitory computer-readable medium storing specific computer-executable instructions that, when executed by a processor, cause a computer system to at least: implement an initial data store and a target data store, migrate a portion of data from the initial data store to the target data store, receive a request to create a first data entity in the initial data store during the migration of the data, determine that the initial data store is designated as a primary data store, determine that the initial store state is open, verify that a key associated with the request exists in the target data store, and upon determining that the key does not exist, create the first data entity in the initial data store. The transaction to create the first data entity can include committing the first data entity to the initial data store or aborting the transaction.

[0015] In some embodiments, the instructions may cause the distributed computing system to: designate the target data store as a primary data store, receive a second request to create a second data entity associated with the second key, determine that the target data store is designated as primary, determine the target store state, verify that the second key does not exist in the initial data store, and in response to these determinations, create the second data entity at the target data source.

[0016] In another embodiment, the instructions may cause the distributed computing system to: create new immutable data in the target data store, designate the initial data store as a primary data store, transfer an additional portion of the data from the target data store to the initial data store, and during the transfer, receive a third request to create a third data entity. The third request may include a third key. The instructions may further cause the distributed computing system to: determine that the initial data store has been designated as the primary data store, determine an initial store state, verify that the third key does not exist in the target data store, and create the third data entity in the initial data store.

[0017] The embodiments may be implemented by using a computer program product comprising computer programs / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure. [Brief description of the drawings]

[0018] [Figure 1] FIG. 2 illustrates a simplified block diagram and associated flowchart of an exemplary process for migrating data from an initial data store to a target data store, in accordance with some embodiments. [Diagram 2] FIG. 1 illustrates an exemplary distributed computing system that implements a workflow service that can initiate processes for transactions with a data store, according to some embodiments. [Diagram 3] FIG. 2 is a simplified block diagram illustrating an exemplary initial data store configuration and a target data store configuration for data migration, according to one embodiment. [Figure 4] 4 is a flow diagram of an example process for handling process invocation logic during migration of data from an initial data store to a target data store, according to some embodiments. [Diagram 5]FIG. 2 illustrates a flowchart of an example process for rolling back a data migration from an initial data store to a target data store, according to some embodiments. [Figure 6] 1 is a simplified flow diagram of an example process for creating a first data entity during migration of data from an initial data store to a target data store, according to some embodiments. [Figure 7] 1 is another simplified flow diagram of an example process for creating a second data entity during migration of data from an initial data store to a target data store, according to some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 9] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 11] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 12] FIG. 1 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting.

[0020] The techniques described herein refer to workflow as a service (WFaaS), where workflow processing tasks (also referred to as workflow instances) may be distributed to one or more workers (e.g., client devices, worker processes, worker threads, etc.) hosted within a distributed computing system (e.g., a client / server environment, a cloud computing environment, etc.). However, the techniques may also be applied to other computing applications that may transact with one or more data stores to create data entities. A distributed computing system may include any suitable number of computing instances (e.g., computing nodes, virtual machines, virtualization containers, hosts, or other suitable physical or virtual computing resources) that perform operations to execute a workflow instance or other operations of a workflow service. A workflow instance may be associated with the creation of one or more data entities. The data entities may include data associated with the workflow instance. After a workflow instance is launched by creating one or more data entities, the workers may execute code (e.g., customer-created code described by the workflow instance data). As a result of a workflow instance, the workflow instance data may be updated. This may include creating or modifying one or more data entities.

[0021] A workflow instance may be initiated by a request (e.g., a customer request). Such a request may not depend on the underlying configuration of a data store of the distributed computing system. For example, a customer request may include information for initiating a workflow instance that does not specify the particular data store or database that the distributed computing system uses. The request may include key data, which may include a key value and a range. The key data may be idempotent so that it can be used to guarantee uniqueness of a workflow instance for a key within a valid range. That is, there can be only one workflow instance corresponding to a key at a time.

[0022] In addition to launching workflow instances, customers may also request that WFaaS store certain types of immutable data. The immutable data may include the workflow definition. The immutable data may be associated with a workflow instance and, once created, is not updated or deleted by the associated workflow instance or other instances. The immutable data may persist because it may be used by subsequent instances or other transactions to the data store. A workflow instance may also be associated with the creation of transient data, which may include data corresponding to one or more tasks of the workflow instance. Transient data may also include lease or allocation data (e.g., historical allocations) for the workflow instance. Transient data may be created and updated within the lifetime of a particular workflow instance. When a workflow instance associated with transient data terminates, the transient data may no longer be needed by any other workflow process. Transient data may be scheduled to be deleted from the data store after a period of time as part of garbage collection or other processes for maintaining the data store.

[0023] Data associated with a workflow instance may be persisted in a data store, database, or other storage system. The data store may be any of a variety of data stores, such as, for example, a relational database, a key-value database, another NoSQL database, a multimodal database, or other databases. The data store may be configured as a pluggable database (PDB), which is a container within a larger data store. In a distributed computing system, data storage requirements of tenants and applications may change (e.g., scalability requirements, customers needing dedicated data stores, additional storage infrastructure, storage load balancing, etc.), which may require migration of data from one data store to another. For example, the data may be persisted in a key-value database. Migration to another target database (e.g., a multimodal database) may be required. To ensure that customers do not experience downtime during the migration, the computing environment may incorporate additional logic to handle customer requests that may cause transactions with the initial or target data store. In this way, customer requests may remain independent of the state of the distributed computing system and the data store implemented during the migration. That is, from the customer's perspective, no downtime occurs because the customer's requests can function the same way before, during, and after the data migration.

[0024] Continuing with the WFaaS example above, migrating a WFaaS customer from one data store to another may include replicating some of the data from the initial data store to the target data store. The migration may exclude workflow transient data from being transferred because that data may only be needed by a workflow instance during its lifetime. That is, the migration may only replicate immutable data from the initial data store to the target data store. Transient data created in the initial data store may eventually be evicted from the initial data store during a garbage collection process following the termination of the associated workflow instance.

[0025] A customer may request the launch of a new workflow instance throughout the migration process. As noted above, the customer's request may not depend on the configuration of the data stores of the distributed computing system. Once the immutable data of the initial data store is replicated to the target data store, a migration may occur that designates the target data store as the primary data store with which the newly launched workflow instance can transact. Because key data may not be replicated in the target data store, a newly launched workflow instance in the target data store may contain keys that are not currently in the target data store but may be in the initial data store. Because WFaaS enforces uniqueness of workflow instance keys (e.g., singleton keys), the workflow instance launch logic may cause the distributed computing system to check for the existence of the key in the initial data store. If the key is stored in the initial data store, the launch request will cause a conflict since an identical workflow instance already exists in the initial data store. The distributed computing system may then reject the workflow instance launch request.

[0026] Most workflow instances may have a relatively short life span such that they may not last longer than the time required to migrate from the initial data store to the target data store. However, some workflow instances may have a relatively long life span. In either case, a workflow instance launched on an initial data store may include an associated process that transacts with the initial data store for the life span of the workflow instance. Similarly, a workflow instance launched on a target data store may have an associated process that transacts with the target data store for the life span of the workflow instance. Thus, after migration to the target data store, the distributed computing system hosting the workflow process may continue to maintain both data stores until all temporary data stored in the initial data store reaches the end of its life span.

[0027] Prior to migrating from the initial data store to the target data store, the customer may request that the distributed computing system (e.g., cloud services, WFaaS, etc.) create new immutable data (e.g., create a new workflow definition). If the initial data store is still the primary data store and will receive the workflow instance to be launched, the new immutable data may be created in both the initial data store and the target data store to ensure that this new immutable data is replicated in both data stores.

[0028] Many advantages are achieved by the techniques described herein. Traditional methods of achieving zero-downtime database migration result in highly complex, inefficient data replication and require manual coordination. For example, real-time data replication requires all data in the initial data store to be replicated to the target data store, which results in unnecessary replication of temporary data in the target data store. In addition, real-time replication requires replication of data created during the migration process such that new data is "replicated" in real-time between both stores as it is created. Real-time replication methods become highly complex when the initial and target data stores are of different types (e.g., key-value and multimodal) with different schemas. As another example, distributed transaction mechanisms that enforce critical constraints between two data stores become highly complex and require additional computing resources to manage the transaction host.

[0029] In contrast, workflow invocation logic allows replicating parts of data without downtime while maintaining key constraints, providing a computationally and time-efficient process for data store migration. Replicating only immutable data minimizes migration transition time because immutable data can be replicated while the initial data store is operational and working properly before migration begins. Furthermore, real-time replication of data during migration from one data store to another is limited to immutable data rather than all data, and occurs only during part of the migration process. Creation of immutable data is typically an infrequently performed operation, reducing the computational cost of replicating immutable data created during migration.

[0030] FIG. 1 illustrates a simplified block diagram 100 and associated flow chart of an exemplary process 101 for migrating data from an initial data store 112 to a target data store 114. The initial data store 112 may be any of several different types of data stores, databases, or other storage systems, including, but not limited to, a key-value database, a NoSQL database, a multimodal database, and the like. Similarly, the target data store 114 may be any of the same type of data store or database. In some embodiments, the target data store 114 is a different type of data store than the initial data store 112. For example, the initial data store 112 may be a key-value database, while the target data store may be a multimodal database. In some other embodiments, the initial data store 112 and the target data store 114 may be the same type of database. According to certain embodiments, the initial data store 112 and the target data store 114 may have the same or different schemas.

[0031] The initial data store 112 and the target data store 114 may be implemented in a distributed computing system, such as a cloud computing environment. The distributed computing system may be configured to provide one or more applications with the ability to transact with the initial data store 112, the target data store 114, or one or more other data stores. The distributed computing system may also be configured to host and run applications for one or more customers as a multi-tenant environment. In a multi-tenant environment, one or more customers may interact with instances of applications running in the distributed computing system, which in turn may transact with one or more data stores, including the initial data store 112 and the target data store 114. In some embodiments, one or both of the data stores may be implemented as pluggable databases. A pluggable database may be a virtual database within a larger virtual or physical database. In some embodiments, each customer using the distributed computing system may have its own pluggable database for each application. For example, a customer may have a database associated with workflow data. In some other embodiments, one or more customers may share one or more pluggable databases associated with the applications. Isolation requirements within a multi-tenant environment may dictate the extent to which data storage resources are shared within the environment.

[0032] Turning now to the flow diagram of process 101, the migration of data from the initial data store 112 to the target data store 114 can be divided into phases represented by blocks of process 101. The phases may include an initial stabilization phase (block 102), a pre-migration phase (block 104), a migration phase (block 106), a post-migration phase (block 108), and a stabilization (termination) phase (block 110). Process 101 can progress through each phase according to satisfying one or more prerequisite conditions in the current phase and then performing one or more actions that result in a phase transition. For example, a prerequisite condition may be that all host nodes in the distributed computing system recognize that the target data store 114 is designated as the primary data store for the application.

[0033] At block 102, an initial data store 112 is implemented. Implementation of the initial data store 112 may include identifying an existing data store from which data migration is performed. The initial data store 112 may be designated as a primary data store. The distributed computing system may identify the primary data store as a data store with which the distributed computing system can transact. The initial data store 112 may also have an initial store state of "open." When the initial data store 112 is open, a request to start an instance associated with the initial data store 112 (e.g., creating a data entity, starting a workflow instance, etc.) is successful and the distributed computing system can successfully transact with the initial data store 112, as indicated by the "Transaction" arrow in FIG. 100. The initial store state and designation as a primary data store may be represented by data stored in the initial data store 112 or another data store associated with the application (e.g., another WFaaS data store).

[0034] Conditions for transitioning from the stable phase may include all hosts (e.g., WFaaS servers) in the distributed computing system recognizing that the initial data store 112 is designated as the primary data store. Transitioning from the stable phase to the pre-migration phase may include populating the target data store 114.

[0035] At block 104, the target data store 114 may be implemented within the distributed computing system. In some embodiments, the target data store 114 may be a new data store or database provisioned by the distributed computing system. The target data store 114 may have a target store state of "open." While the target store state is open, requests to launch instances (e.g., launch a workflow instance and create data entities) are routed to the initial data store 112 as the primary data store.

[0036] Conditions for transitioning from the pre-migration phase to the transition phase may include that the target data store 114 is properly provisioned, the target store state is “open”, and a portion of the data stored in the initial data store 112 is replicated to the target data store 114. When migrating workflow data, the portion of data replicated may be all immutable data in the initial data store 112, without replicating temporary or key data. Transitioning from the pre-migration phase to the transition phase may include an operation of setting the initial store state to “closed”. This operation may be a transaction with the initial data store 112 to update the initial store state stored in the initial data store 112.

[0037] As briefly mentioned above, during the pre-migration phase, the customer may launch new instances after replicating some of the data or while replicating but before closing the initial data store 112. Because the initial data store 112 is designated as the primary data store, these new instances may have associated processes configured to transact with the initial data store 112 and may create new data in the initial data store 112 that should be replicated in the target data store 114 but may be omitted when the portion of the data is replicated. To ensure proper data replication during the pre-migration phase, transactions that create new data may be configured to create that new data in both the initial data store 112 and the target data store 114 (similar to real-time replication of newly created data). Within the WFaaS system illustrated herein, the new data may be new immutable data.

[0038] At block 106, the migration phase is entered by changing the initial store state from open to closed. If the initial store state is closed, then the launch request will be rerouted from the initial data store 112 to the target data store 114. This rerouting of the launch request may occur even if the initial data store 112 is still designated as the primary data store. Previously launched instances may still have associated processes running and transacting on the initial data store 112. For workflow instances launched on the target data store 114, when the migration process reaches the migration phase, the distributed computing system may check the key data on the initial data store for key idempotence. This check may prevent multiple workflow instances with the same key from being launched at the same time.

[0039] A condition for transitioning from the migration phase to the post-migration phase may be that all hosts (e.g., WFaaS servers) recognize that the state of the primary data store (initial data store 112) is “closed.” This condition may require only an acknowledgement from the hosts in the computing environment, so the migration phase takes a relatively short time (e.g., on the order of a few minutes). Transitioning from the migration phase to the post-migration phase may include designating the target data store 114 as the primary data store.

[0040] At block 108, the target data store 114 now becomes the primary data store for instances launched within the distributed computing system. As the primary data store, new launch requests are routed to the target data store 114. As with the migration phase of block 106, previously launched instances may still have associated processes running and transacting with the initial data store 112.

[0041] A condition for transitioning from the post-migration phase to the stable (termination) phase may include data in the initial data store 112 that was not replicated to the target data store 114 being drained from the initial data store 112. In the case of workflow data, the drained data is temporary data and may be deleted by a garbage collection process after a period of time. A further condition may be that all keys stored in the initial data store 112 go out of scope. The post-migration phase may last for a relatively long period of time (e.g., several days) because data from the initial data store 112 may persist for the life of the instance and its associated processes that run and transact on the initial data store 112. The transition to the stable (termination) phase may be removal (e.g., deprovisioning) of the initial data store 112 from the distributed computing system.

[0042] Finally, in block 110, the computing environment returns to a stable state in the Stable (End) phase. The target data store 114 is the primary data store and can receive requests to launch instances (e.g., workflow instances, creation of new data entities, etc.). The Stable (End) phase is a mirror of the Initial Stable phase, with the target data store 114 taking the place of the Initial data store 112.

[0043] As described in more detail below with respect to FIG. 5, the migration process may include the ability to support rollback of the data migration. In the case of a rollback operation, the initiation logic of the workflow instance remains the same, providing the techniques disclosed herein with additional flexibility and advantages over conventional migration methods. Rollback of the migration from the stable (finished) phase proceeds as described above for the forward migration. Rollback from the pre-migration phase includes deleting the target data store 114. In some embodiments, to avoid conflicts, rollback may not be allowed during the migration phase. Rollback from the post-migration phase may include replicating new data created in the target data store 114 to the initial data store 112, and then proceeding as described above as if the system was in the pre-migration state of the forward migration, but with the "initial" and "target" nature of the data stores swapped.

[0044] FIG. 2 illustrates a distributed computing system 210 in a cloud computing environment 200 that implements a workflow service that can initiate processes to transact with a data store, according to some embodiments. The distributed computing system 210 may be implemented by one or more computing systems that execute computer-readable instructions (e.g., code, programs) to implement the distributed computing system. As illustrated in FIG. 2, the distributed computing system 210 includes various systems, including a load balancer 212, a gateway 214 (e.g., a multi-tenant gateway), an application programming interface (API) server 216, and one or more execution environments 222. Portions of data or information used by or generated by the system illustrated in FIG. 2 may be stored in a data store 256. In some embodiments, the data store 256 may include one or more data stores described herein, including, for example, the initial data store 112 and the target data store 114 of FIG. 1. The system illustrated in FIG. 2 may be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a computing system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (eg, a memory device).

[0045] Distributed computing system 210 can be implemented in a variety of configurations. In the embodiment shown in FIG. 2, distributed computing system 210 can be implemented on one or more servers of a cloud provider network. The computing environment 200 including distributed computing system 210 shown in FIG. 2 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, distributed computing system 210 can be implemented using more or fewer systems than those shown in FIG. 2, may combine two or more systems, or may have a different configuration or arrangement of systems.

[0046] In some embodiments, an execution environment (e.g., execution environment 222) may represent an environment (e.g., a “client” node, a host node, etc.) for executing some or all of the tasks (e.g., workflow instances, step instances, etc.) of a workflow. Different execution environments may be associated with tenants (e.g., customers). For example, in the embodiment shown in FIG. 2, execution environment 222 is associated with tenant 220 of the distributed computing system. One or more different execution environments may be associated with additional tenants. An execution environment may be configured to utilize any suitable number of computing nodes to perform operations in a coordinated manner. A “computing node” (also referred to herein as a “node”) may include a server, a computing device, a virtual machine, or any suitable physical or virtual computing resource configured to perform operations as part of the execution environment.

[0047] Resources allocated to tenants of a distributed computing system are selected from a plurality of cloud-based resources arranged hierarchically. For example, as shown in FIG. 2, the resources may be a pool of cloud infrastructure services 240. Cloud infrastructure services 240 may include a metadata store 242, workflow as a service (WFaaS) 244, a virtual cloud network (VCN) 246, identity and access management (IAM) services 248, and a data catalog 250. Metadata store 242 may be implemented as a key-value database as a service (KaaS) or other suitable data storage service.

[0048] In particular embodiments, users (e.g., customers 202) may interact with the distributed computing system 210 via computing devices 204 that are communicatively coupled to the distributed computing system 210, possibly via a public network 208 (e.g., the Internet). The computing devices 204 may be of various types, including, but not limited to, mobile phones, tablets, desktop computers, etc. The users may interact with the cloud computing system using a console user interface (UI) (which may be a graphical user interface) of an application executed by the computing device or via API operations provided by the distributed computing system 210.

[0049] As an example, a user associated with a tenant 220 of the distributed computing system 210 may interact with the distributed computing system 210 by sending a request (e.g., launch request, launch instance request, etc.) to the distributed computing system 210 to create a new data entity (e.g., launch a workflow instance). In response, the distributed computing system 210 may execute one or more workers 224 within the execution environment 222. The launch request may be received by a load balancer 212 within the distributed computing system 210, which may send the request to a multi-tenant proxy service within the distributed computing system, such as a gateway 214. The gateway 214 may be responsible for authenticating / authorizing the user's request and routing the request to an API server 216, which may be configured to communicate with one or more cloud services (e.g., metadata store 242, WFaaS 244) or the execution environment 222 to perform operations to fulfill the request. In a particular example, the gateway 214 may represent a shared multi-tenant Hypertext Transfer Protocol (HTTP) proxy service that authorizes users and sends their requests to the API server 216.

[0050] For a WFaaS process, the worker 224 can execute one or more tasks corresponding to one or more workflow instances. The worker 224 can be configured to transact with the data store 256 and the metadata store 242. The metadata store 242 can identify a designated primary data store. The one or more tasks corresponding to the workflow instance can include creating one or more data entities in the data store 256. The interaction with the metadata store 242 can be via the data access object 226, which can be configured to interact with a data storage service present in the distributed computing system 210. FIG. 2 does not constrain the data storage system that implements the data store 256.

[0051] The launch request may include a key. In some embodiments, the key may be a surrogate key with a surrogate scope. WFaaS 244 may respond to the launch request by returning a globally unique instance ID to the customer. The instance ID may be an example of temporary data, while the key is an example of key data. When a workflow instance is launched, the instance ID and key may be stored in a data store with which the worker 224 is configured to transact.

[0052] In some embodiments, customers can query workflow instances by sending queries that target the instance ID or surrogate key associated with the workflow instance. To protect this functionality during migration, the instance ID can include a reference or other identifying information of the data store with which the instance was associated at launch. Queries that target the instance ID then target the data store that is handling the instance. Encoding the data store identifying information into the instance ID allows associated program functions (e.g., API calls) to successfully interact with the workflow instance during migration. For surrogate key queries, WFaaS targets both the initial and target data stores and receives query results (e.g., temporary data for the instance) from the data store that contains the instance.

[0053] FIG. 3 is a simplified diagram 300 of an exemplary initial data store configuration 302 and a target data store configuration 304 for data migration, according to one embodiment. Both the initial data store configuration 302 and the target data store configuration 304 are implemented for a multi-tenant environment running workflow instances, with each tenant associated with its own database. As shown in FIG. 3, the initial data store configuration 302 can include one or more key-value databases provided by a database service (KaaS) 306. For example, the tenant databases 308-312 can be dedicated databases (e.g., pluggable databases) for each tenant's workflow data (e.g., immutable data, temporary data, and key data). In this example, the database service (KaaS) 306 plays a dual role as both a metadata store 314 (e.g., metadata store 242) and a data storage device (e.g., data storage device 256). Additionally, the initial data store configuration 302 can include one or more additional data stores or databases (e.g., metadata store 314) implemented as key-value databases. The metadata store 314 may be used to store data associated with the control plane service 316 of the workflow service (eg, WFaaS, which may be similar to service (WFaaS) 244 of FIG. 2).

[0054] The workflow services data plane 318 may represent data plane services of the WFaaS used by one or more tenants of a distributed computing system, including the execution environment 222 described with respect to FIG. 2. Thus, the workflow services data plane 318 may include execution processes associated with the workflow instances of the tenants of the distributed computing system. The workflow services data plane 318 may interact with the database service (KaaS) 306 through an initial data access object (DAO) 322. The initial DAO 322 may be an interface layer configured for a particular database service, in this case KaaS 306. Similar to data access objects typical in the art, the initial DAO 322 may provide an abstraction between the execution processes of the workflow data plane 318 (e.g., worker 224) and the data storage system used by KaaS 306. In some other embodiments, the initial data store configuration may implement different types of database services (e.g., multimodal) that may use different DAOs to provide an abstraction layer.

[0055] WFaaS may also include one or more background jobs 320 to help manage the execution of workflow instances. These background jobs 320 may include an anti-entropy engine, a delayed workflow execution process, garbage collection, and service health monitoring. These jobs may be configured to transact with one or more data stores in the initial data store configuration 302. The background jobs 320 may interface with KaaS 306 using the same initial DAO 322 as for the workflow service data plane 318.

[0056] After data migration, the target data store configuration 304 may include one or more data stores that are different from the data stores in the initial data store configuration 302. As shown in FIG. 3, the target data store configuration 304 may include a container database 324. The container database 324 may include one or more pluggable databases that include workflow data plane data of the tenants migrated from the initial data store configuration. The new configuration after migration may include data stored in separate data stores for each customer. In some embodiments, a tenant may have its own data stores, e.g., tenant database 326 and tenant database 328. Other tenants of the distributed computing system may have one or more pluggable databases for tenant data store 330.

[0057] Because the target data store configuration 304 may include a different data storage system than the initial data store configuration 302, in some embodiments, the workflow services data plane 318 may implement a target DAO 332 that is configured to interface with the data storage system of the target data store configuration 304. In embodiments where the type of data storage system implemented for the post-migration configuration has not changed from the pre-migration configuration, the DAOs used by the workflow services may not be changed. In some embodiments, one or more data stores of the initial data store configuration 302 may not be migrated. For example, the metadata store 314 of the workflow services control plane 316 data may remain unchanged during the tenant migration process. The metadata store 314 may be of a different type (e.g., key-value) than other data stores in the target data store configuration 304. To interface with the metadata store (via KaaS 306), the workflow services data plane 318 may retain the initial DAO 322 in addition to implementing the new target DAO 332.

[0058] FIG. 4 is a flow diagram of an exemplary process for handling instance startup logic during migration of data from an initial data store to a target data store, according to some embodiments. The initial data store may be one of the initial data stores described herein, including the initial data store 112 of FIG. 1. Similarly, the target data store may be one of the target data stores described herein, for example, the target data store 114. Process 400 (as well as processes 600 and 700) is illustrated as a logical flow diagram, each operation of which represents a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be omitted or the process may be implemented in any order and / or in any combination in parallel.

[0059] The initiation logic illustrated by process 400 may be implemented by a process of a distributed computing system responsible for reading and writing to a data store within the distributed computing system. The reading and writing may be performed for the purpose of initiating a workflow instance. In some embodiments, the process may be a component of a workflow service (e.g., WFaaS 244).

[0060] At entry 402, the process of booting an instance during migration begins by determining a primary data store. Depending on the phase of the migration process, the primary data store can be an initial data store or a target data store. For example, during the pre-migration and migration phases described with respect to FIG. 1, the primary data store is the initial data store. During the post-migration and stable (termination) phases, the primary data store is the target data store. The boot logic of process 400 is migration phase independent and therefore can be applied during any phase of migration or rollback operation between two data stores. A data store that is not designated as a primary data store may be referred to as a secondary data store. The designation of the primary data store may be a flag or other data stored in the primary data store or another data store implemented in the distributed computing system.

[0061] At decision 404, the primary data store determined at 402 is checked to determine its store state. This check can be a transaction with the primary data store to read the store state that may be stored in the primary data store. In some embodiments, the check includes a transaction to read store state data from the different data stores, since the different data stores may contain store state for the primary data store and for the secondary data store. If the primary data store is "open", the launch request is routed to the primary data store. If the primary data store is "closed", the launch request is routed to the secondary data store. Before launching the instance, additional checks can be performed to ensure idempotence of key data and that the primary data store state has not changed within the time required to process the launch request.

[0062] After verifying that the primary data store is open, and that the workflow instance does not violate the singleton key requirements of the workflow service, the distributed computing system may check the secondary data store for the presence of the key received with the launch request, in decision 406. If the secondary data store has a matching key, an instance with that key has already been launched in the secondary data store. A second instance with the same key may not be able to be launched, so a conflict may occur at endpoint 408 and the process may be aborted. This behavior is the same during the stable phase, when no migration is occurring. A customer cannot launch a second instance using the same key.

[0063] If there is no matching key in the secondary data store, the process may proceed to block 410, where the instance is launched with a corresponding task to be executed by an execution process (e.g., worker 224) in the execution environment and configured to conduct transactions with the data store currently designated as the primary data store, which may be either the initial data store or the target data store, depending on the migration phase.

[0064] During the transition, the primary data store may have its store state changed from "open" to "closed". Because the change in store state may be transactional with the primary database, the state may be updated (and the update committed) in the primary data store before the instance is fully launched (e.g., before the workflow instance is committed to the primary data store). In other words, the execution of the launch request and the receipt of the transition instruction from one transition phase to the next may occur almost simultaneously, so that the primary data store is closed before the instance is launched and its associated key data is stored in the corresponding data store. In this case, a second launch request with the same key may be received and routed to the secondary data store (because the primary data store has just been closed) and the second instance may be successfully launched in the secondary data store (because the key of the first launch request has not been committed to the primary data store). To avoid this, before committing the launch on the primary data store, a second check is made in decision 412 to check whether the primary data store is still "open". If not, the process may be aborted with an exception at endpoint 414.

[0065] If the primary data store is still open, the launch transaction may be committed at block 416. A final check may be performed at decision 418 to ensure that the launch transaction (e.g., committing the instance metadata to the primary data store) was successful. If so, the process may terminate with a report of success at endpoint 420. If the launch commit was not successful, the process may be aborted at endpoint 408.

[0066] Returning to decision 404, if the primary data store is closed when the launch request is received, the request is routed to the secondary data store. Blocks 422, 424, 426, 428, and 430 are similar to the operations described for corresponding blocks 406, 408, 410, 416, and 418 described above, respectively, except that an instance is launched on the secondary data store, so that the running process corresponding to the instance is configured to transact with the secondary data store. At decision 422, a check of key data is performed on the primary data store (e.g., the initial data store if the secondary data store is currently the target data store). In the migration process, the state of the primary data store is changed from "open" to "closed" before designating the secondary data store as the new primary data store, so the safety check operation of decision 412 may not be performed on the secondary data store.

[0067] FIG. 5 illustrates a flow diagram 500 of an example process 501 for rolling back a migration of data from an initial data store to a target data store, according to some embodiments. The initial data store and the target data store may be similar to the initial data store 112 and the target data store 114 of FIG. 1. The process 101 may be the same as the process illustrated in FIG. 1, with five phases of data migration. Rolling back a migration may include replicating new data created in the target data store, closing the target data store, designating the initial data store as the primary data store, and finally deleting the target data store from the distributed computing system. In general, the rollback operation is the reverse of the forward migration operation described above.

[0068] In some embodiments, rollback operations are supported from all phases of the migration process except the migration phase. As briefly mentioned above, a rollback operation 502 from the pre-migration phase (i.e., before the initial data store is closed) may include deleting (e.g., de-provisioning) the target data store since no new data has been created in the target data store. A rollback operation 504 from the stable (termination) phase may be the reverse of the forward migration of process 101 and may be treated as a separate migration rather than a rollback.

[0069] In the case of a rollback process 501 from the post-migration phase, the target data store may include new data (e.g., new immutable data) that is created by the distributed computing system during the post-migration phase (e.g., in response to the creation of a new workflow definition by a customer). The rollback proceeds similarly to the forward migration, but the process is reversed.

[0070] The pre-rollback phase is entered by changing the initial store state from "closed" to "open" at block 506. As a condition of entering the pre-rollback phase, some of the new data in the target data store (e.g., new immutable data) is replicated to the initial data store.

[0071] At block 508, the rollback phase is entered by closing the target data store. The invocation logic described with respect to Figure 4 continues to apply to invocation requests received in the pre-rollback, rollback, and post-rollback phases. A condition for transitioning to the post-rollback phase may be that all hosts (e.g., WFaaS servers) recognize that the state of the primary data store is closed.

[0072] At block 510, the post-rollback phase is entered by designating the initial datastore as the primary datastore. A condition for entering the stable phase may be that all hosts (e.g., WFaaS servers) recognize that the initial datastore has been designated the primary datastore and that data (e.g., temporary data, key data) is drained from the target datastore. Once these conditions are met, the stable phase may be entered including only the initial datastore by deleting the target datastore.

[0073] 6 is a simplified flow diagram of an example process 600 for creating a data entity (e.g., initiating a workflow instance) during migration of data from an initial data store to a target data store, according to some embodiments. The initial data store and the target data store may be similar to any of the initial and target data stores described herein, such as the initial data store 112 and the target data store 114 of FIG. 1.

[0074] Process 600 may begin at block 602 with the distributed computing system implementing an initial data store. Implementing the initial data store may include provisioning the data store. The initial data store may be configured to store data associated with a user (e.g., customer 202) of the initial data store. The user may be a tenant of the distributed computing system. The initial data store may have an initial store state. The initial store state may be either open or closed. In some embodiments, the data associated with the user may be immutable data, temporary data, and key data. The initial data store may be designated as a primary data store. The designation as a primary data store and the initial store state may be data values ​​stored in the initial data store or another data store associated with the distributed computing system.

[0075] At block 604, the distributed computing system can implement a target data store. Like the initial data store, the target data store can be provisioned from resources available in the distributed computing environment. The target data store can be the same type of data store (e.g., the same database type) as the initial data store, or it can be a different type. The target data store can have a target store state. Like the initial store state, the target store state can be open or closed and can be a data value stored in the target data store or another data store in the distributed computing environment.

[0076] At block 606, a data migration process may be initiated for a portion of the data stored in the initial data store. The migration process may be similar to process 101 described with respect to FIG. 1. In some embodiments, the migration process migrates a tenant of the WFaaS system from the initial data store corresponding to the tenant's workflow data to the target data store. The portion of the data may include immutable workflow data, while the data that is not migrated may include temporary and key data associated with one or more workflow instances. Migrating the portion of the data may include replicating the portion from the initial data store to the target data store. The migration of the portion of the data may continue until the temporary and key data is drained from the initial data store.

[0077] During the migration of some of the data, the distributed computing system can still handle requests to launch instances. Data entities associated with the instances can be created in either the initial data store or the target data store according to the phase of the migration and the processes described herein. At block 608, the distributed computing system (e.g., a scheduling process of the service (WFaaS) 244) can receive a request to create a first data entity (e.g., a workflow instance including a task to be executed by the worker 224). The request can include metadata associated with the first data entity, including a key and a key range. One or more processes (e.g., the worker 224) of the distributed computing system can be configured to transact with the initial data store to create the first data entity. For example, the distributed computing system can read immutable data (e.g., a workflow definition) to execute the workflow.

[0078] At block 610, the distributed computing system may determine that an initial data store is designated as a primary data store in response to the request. At block 612, an initial store state associated with the initial data store may also be determined. Determining the initial store state may include a transaction with the initial data store to read a data value corresponding to the state. The initial store state may be either "open" or "closed."

[0079] In response to determining the initial store state, the distributed computing system may verify that the key does not exist in the target data store, at block 614. The distributed computing system may create the first data entity in the initial data store, at block 616. Creating the first data entity in the initial data store may include writing, reading, updating, and deleting data in the initial data store. The distributed computing system may also abort the creation of the first data entity before committing data associated with the first data entity to the initial data store.

[0080] 7 is another simplified flow diagram of an example process 700 for creating a second data entity (e.g., launching a second workflow instance) during migration of data from an initial data store to a target data store, according to some embodiments. As with FIG. 6, the initial data store and the target data store may be similar to other initial data stores and target data stores described herein. Process 700 may be considered an extension of process 600 discussed above.

[0081] The process begins at block 702 during the migration of data from an initial data store to a target data store. The target data store may be designated as a primary data store. The distributed computing system may receive a second request to create a second data entity at block 704. The second request may include metadata associated with the second data entity, including a second key and a second key range. One or more processes (e.g., workers 224) of the distributed computing system may be configured to transact with the target data store to create the second data entity. For example, the second distributed computing system may read the immutable data (e.g., a workflow definition) to execute a workflow.

[0082] At block 706, the distributed computing system may determine that the target data store has been designated the primary data store in response to the request (block 702). At block 708, the target store state may also be determined. The target store state may be "open" or "closed." In response to determining that the target store state is open, the distributed computing system may verify that the second key does not exist in the initial data store at block 710. Finally, at block 712, the distributed computing system may create a second data entity in the target data store. Creating the second data entity in the target data store may include writing, reading, updating, and deleting data in the target data store. The distributed computing system may also abort the creation of the second data entity before committing data associated with the second data entity to the target data store.

[0083] Example Infrastructure as a Service Architecture As mentioned above, Infrastructure as a Service (IaaS) is one particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud computing providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provide various services (e.g., billing, monitoring, logging, load balancing, and clustering, etc.) that accompany these infrastructure components. Thus, these services can be policy-driven, so that IaaS users can potentially implement policies that drive load balancing to maintain application availability and performance.

[0084] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet, and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform and create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions such as balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0085] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but does not have to be, a third-party service that specializes in providing (e.g., providing, renting, selling) IaaS. An entity may also choose to deploy a private cloud and become its own infrastructure service provider.

[0086] In some examples, IaaS deployment is the process of placing a new application, or a new version of an application, onto a prepared application server, etc. This may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling things like (OS), middleware, and / or application deployment (e.g., self-service virtual machines (e.g., that can be spun up on demand)).

[0087] In some instances, IaaS provisioning may even refer to obtaining computers or virtual hosts for use and installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, so provisioning may need to be performed first.

[0088] In some cases, there are two distinct challenges with IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything can run. Second, there is the challenge of evolving the existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how each works together, etc.) can be described declaratively. In some cases, once the topology is defined, workflows can be generated to create and / or manage the various components described in the configuration files.

[0089] In some examples, the infrastructure may have many elements that are interconnected. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shared computing resources), also known as a core network. In some examples, there may also be one or more inbound / outbound traffic group rules and one or more virtual machines (VMs) that are provisioned to define how the network's inbound and / or outbound traffic is configured. Other infrastructure elements such as load balancers, databases, etc. may also be provisioned. The infrastructure may evolve in stages as more infrastructure elements are desired or added.

[0090] In some cases, continuous deployment techniques may be used to enable deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques enable infrastructure management within these environments. In some examples, a service team may write code that is desirable to deploy to one or more, but often many, different production environments (e.g., across various different geographic locations, possibly even across the world). However, in some examples, it may be necessary to first set up the infrastructure to which the code will be deployed. In some cases, provisioning may be done manually, and a provisioning tool may be utilized to provision resources and / or a deployment tool may be utilized to deploy the code after the infrastructure has been provisioned.

[0091] FIG. 8 is a block diagram 800 illustrating an example IaaS architecture pattern, according to at least one embodiment. A service operator 802 can be communicatively coupled to a secure host tenant 804, which can include a virtual cloud network (VCN) 806 and a secure host subnet 808. In some examples, the service operator 802 can use one or more client computing devices, which can be portable handheld devices (e.g., iPhone, mobile phone, iPad, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass head mounted display), running software such as Microsoft Windows Mobile, and / or various mobile operating systems such as iOS, WindowsPhone, Android, BlackBerry8, PalmOS, and Internet, email, short message service (SMS), Blackberry, or other communication protocols in effect. Alternatively, the client computing devices can be general purpose personal computers, including, for example, personal computers and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing device may be a workstation computer running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively, or in addition, the client computing device may be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device capable of communicating over a network with access to the VCN 806 and / or the Internet.

[0092] The VCN 806 may include a local peering gateway (LPG) 810 that may be communicatively coupled to a secure shell (SSH) VCN 812 via an LPG 810 that is included in the SSH VCN 812. The SSH VCN 812 may include an SSH subnet 814, which may be communicatively coupled to a control plane VCN 816 via an LPG 810 that is included in the control plane VCN 816. The SSH VCN 812 may also be communicatively coupled to a data plane VCN 818 via the LPG 810. The control plane VCN 816 and the data plane VCN 818 may be included in a service tenant 819, which may be owned and / or operated by the IaaS provider.

[0093] The control plane VCN 816 may include a control plane demilitarized zone (DMZ) tier 820 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers have limited responsibility and may help prevent intrusions. Additionally, the DMZ tier 820 may include one or more load balancer (LB) subnets 822, a control plane app tier 824 that may include an app subnet 826, and a control plane data tier 828 that may include a database (DB) subnet 830 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 822 included in the control plane DMZ tier 820 may be communicatively coupled to the app subnet 826 included in the control plane app tier 824 and an Internet gateway 834 that may be included in the control plane VCN 816, and the app subnet 826 may be communicatively coupled to the DB subnet 830 included in the control plane data tier 828, as well as a service gateway 836 and a network address translation (NAT) gateway 838. The control plane VCN 816 may include a service gateway 836 and a NAT gateway 838 .

[0094] The control plane VCN 816 can include a data plane mirrored app layer 840 that can include an app subnet 826. The app subnet 826 included in the data plane mirrored app layer 840 can include a virtual network interface controller (VNIC) 842 on which a compute instance 844 can run. The compute instance 844 can communicatively couple the app subnet 826 of the data plane mirrored app layer 840 to the app subnet 826 that can be included in the data plane app layer 846.

[0095] The data plane VCN 818 can include a data plane app layer 846, a data plane DMZ layer 848, and a data plane data layer 850. The data plane DMZ layer 848 can include a LB subnet 822, which can be communicatively coupled to an app subnet 826 of the data plane app layer 846 and an Internet gateway 834 of the data plane VCN 818. The app subnet 826 can be communicatively coupled to a service gateway 836 of the data plane VCN 818 and a NAT gateway 838 of the data plane VCN 818. The data plane data layer 850 can also include a DB subnet 830, which can be communicatively coupled to the app subnet 826 of the data plane app layer 846.

[0096] The Internet gateways 834 of the control plane VCNs 816 and the data plane VCNs 818 may be communicatively coupled to a metadata management service 852, which may be communicatively coupled to the public Internet 854. The public Internet 854 may be communicatively coupled to NAT gateways 838 of the control plane VCNs 816 and of the data plane VCNs 818. The service gateways 836 of the control plane VCNs 816 and of the data plane VCNs 818 may be communicatively coupled to cloud services 856.

[0097] In some examples, a service gateway 836 in a control plane VCN 816 or a data plane VCN 818 can make application programming interface (API) calls to cloud services 856 without traversing the public Internet 854. API calls from a service gateway 836 to a cloud service 856 can be one-way: the service gateway 836 can make an API call to the cloud service 856, and the cloud service 856 can send the requested data to the service gateway 836. However, the cloud service 856 may not be able to initiate an API call to the service gateway 836.

[0098] In some examples, the secure host tenant 804 can be directly connected to the service tenant 819 or can be otherwise separate. The secure host subnet 808 can communicate with the SSH subnet 814 through the LPG 810, which can allow bidirectional communication through otherwise separate systems. Connecting the secure host subnet 808 to the SSH subnet 814 can give the secure host subnet 808 access to other entities in the service tenant 819.

[0099] The control plane VCN 816 can enable users of a service tenant 819 to set up or provision desired resources. The desired resources provisioned in the control plane VCN 816 can be deployed or used in the data plane VCN 818. In some examples, the control plane VCN 816 can be separate from the data plane VCN 818, and the data plane mirror app layer 840 of the control plane VCN 816 can communicate with the data plane app layer 846 of the data plane VCN 818 via a VNIC 842, which can be included in the data plane mirror app layer 840 and the data plane app layer 846.

[0100] In some examples, a user or customer of the system may make a request, such as, for example, a create, read, update, or delete (CRUD) operation, via the public internet 854, which may communicate the request to a metadata management service 852. The metadata management service 852 may communicate the request to the control plane VCN 816 via an internet gateway 834. The request may be received by a LB subnet 822 included in the control plane DMZ layer 820. The LB subnet 822 may determine that the request is valid, and in response to this determination, the LB subnet 822 may send the request to an app subnet 826 included in the control plane app layer 824. If the request is confirmed and a call to the public internet 854 is required, the call to the public internet 854 may be sent to a NAT gateway 838, which may make the call to the public internet 854. Memory that may be desirable to be stored with the request may be stored in the DB subnet 830.

[0101] In some examples, the data plane mirror app layer 840 can facilitate direct communication between the control plane VCN 816 and the data plane VCN 818. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 818. Through the VNIC 842, the control plane VCN 816 can communicate directly with the resources included in the data plane VCN 818, thereby enabling it to perform configuration changes, updates, or other suitable modifications to the resources included in the data plane VCN 818.

[0102] In some embodiments, the control plane VCN 816 and the data plane VCN 818 can be included in the service tenant 819. In this case, a user or customer of the system may not own or operate either the control plane VCN 816 or the data plane VCN 818. Instead, an IaaS provider may own or operate the control plane VCN 816 and the data plane VCN 818, both of which may be included in the service tenant 819. This embodiment may allow for network isolation that may prevent a user or customer from interacting with other users' or other customers' resources. This embodiment also allows a user or customer of the system to store databases privately without having to rely on the public Internet 854, which may not have the desired level of threat protection for storage.

[0103] In another embodiment, the LB subnet 822 included in the control plane VCN 816 may be configured to receive signals from the service gateway 836. In this embodiment, the control plane VCN 816 and the data plane VCN 818 may be configured to be called by the IaaS provider's customers without calling the public Internet 854. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in a service tenant 819 that is controlled by the IaaS provider and may be isolated from the public Internet 854.

[0104] 9 is a block diagram 900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 902 (e.g., service operator 802 of FIG. 8 ) can be communicatively coupled to a secure host tenant 904 (e.g., secure host tenant 804 of FIG. 8 ), which can include a virtual cloud network (VCN) 906 (e.g., VCN 806 of FIG. 8 ) and a secure host subnet 908 (e.g., secure host subnet 808 of FIG. 8 ). The VCN 906 can include a local peering gateway (LPG) 910 (e.g., LPG 810 of FIG. 8 ), which can be communicatively coupled to a secure shell (SSH) VCN 912 (e.g., SSH VCN 812 of FIG. 8 ) via the LPG 810 included in the SSH VCN 912. The SSH VCN 912 can include an SSH subnet 914 (e.g., SSH subnet 814 in FIG. 8 ), which can be communicatively coupled to a control plane VCN 916 (e.g., control plane VCN 816 in FIG. 8 ) via an LPG 910 included in the control plane VCN 916. The control plane VCN 916 can be included in a service tenant 919 (e.g., service tenant 819 in FIG. 8 ), and the data plane VCN 918 (e.g., data plane VCN 818 in FIG. 8 ) can be included in a customer tenant 921, which can be owned or operated by a user or customer of the system.

[0105] The control plane VCN 916 may include a control plane DMZ layer 920 (e.g., control plane DMZ layer 820 of FIG. 8 ) that may include a LB subnet 922 (e.g., LB subnet 822 of FIG. 8 ), a control plane app layer 924 (e.g., control plane app layer 824 of FIG. 8 ) that may include an app subnet 926 (e.g., app subnet 826 of FIG. 8 ), and a control plane data layer 928 (e.g., control plane data layer 828 of FIG. 8 ) that may include a database (DB) subnet 930 (e.g., similar to DB subnet 830 of FIG. 8 ). The LB subnet 922 included in the control plane DMZ tier 920 may be communicatively coupled to an app subnet 926 included in the control plane app tier 924 and to an Internet gateway 934 (e.g., Internet gateway 834 in FIG. 8 ) that may be included in the control plane VCN 916, and the app subnet 926 can be communicatively coupled to a DB subnet 930, a service gateway 936 (e.g., service gateway in FIG. 8 ) and a network address translation (NAT) gateway 938 (e.g., NAT gateway 838 in FIG. 8 ) included in the control plane data tier 928. The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0106] The control plane VCN 916 may include a data plane mirror app layer 940 (e.g., data plane mirror app layer 840 of FIG. 8 ), which may include an app subnet 926. The app subnet 926 included in the data plane mirror app layer 940 may include a virtual network interface controller (VNIC) 942 (e.g., VNIC 842), which may run a compute instance 944 (e.g., similar to compute instance 844 of FIG. 8 ). The compute instance 944 may facilitate communication between the app subnet 926 and the app subnet 926 of the data plane mirror app layer 940, which may be included in the data plane app layer 946 (e.g., data plane app layer 846 of FIG. 8 ), via the VNIC 942 included in the data plane mirror app layer 940 and the VNIC 942 included in the data plane app layer 946.

[0107] An internet gateway 934 included in the control plane VCN 916 can be communicatively coupled to a metadata management service 952 (e.g., metadata management service 852 in FIG. 8), which can be communicatively coupled to the public internet 954 (e.g., public internet 854 in FIG. 8). The public internet 954 can be communicatively coupled to a NAT gateway 938 included in the control plane VCN 916. A service gateway 936 included in the control plane VCN 916 can be communicatively coupled to cloud services 956 (e.g., cloud services 856 in FIG. 8).

[0108] In some examples, the data plane VCN 918 may be included in a customer tenant 921. In this case, the IaaS provider may provide a control plane VCN 916 for each customer, and the IaaS provider may set up a unique compute instance 944 for each customer, which is included in the service tenant 919. Each compute instance 944 may enable communication between the control plane VCN 916 included in the service tenant 919 and the data plane VCN 918 included in the customer tenant 921. The compute instance 944 may enable resources provisioned in the control plane VCN 916 included in the service tenant 919 to be deployed or otherwise used in the data plane VCN 918 included in the customer tenant 921.

[0109] In another example, a customer of the IaaS provider may have a database that resides in a customer tenant 921. In this example, the control plane VCN 916 may include a data plane mirror app layer 940, which may include an app subnet 926. The data plane mirror app layer 940 may reside in the data plane VCN 918, but the data plane mirror app layer 940 may not reside in the data plane VCN 918. That is, the data plane mirror app layer 940 may be accessible to the customer tenant 921, but the data plane mirror app layer 940 may not reside in the data plane VCN 918 or be owned or operated by the IaaS provider customer. The data plane mirror app layer 940 may be configured to make calls to the data plane VCN 918, but may not be configured to make calls to any entities included in the control plane VCN 916. A customer may desire to deploy or use resources in the data plane VCN 918 that are provisioned in the control plane VCN 916, and the data plane mirror app layer 940 may facilitate the desired deployment or other use of the customer's resources.

[0110] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 918. In this embodiment, the customer can determine what the data plane VCN 918 can access, and the customer can limit access from the data plane VCN 918 to the public Internet 954. The IaaS provider may not be able to apply filters or control the data plane VCN 918's access to any external networks or databases. Applying customer filters and controls to the data plane VCN 918 contained in the customer tenant 921 can help isolate the data plane VCN 918 from other customers and from the public Internet 954.

[0111] In some embodiments, cloud services 956 may be called by service gateway 936 to access services that may not exist on the public internet 954, on the control plane VCN 916, or on the data plane VCN 918. The connection between cloud services 956 and the control plane VCN 916 or the data plane VCN 918 may not exist or be continuous. Cloud services 956 may exist on another network owned or operated by the IaaS provider. Cloud services 956 may be configured to receive calls from service gateway 936 or may be configured not to receive calls from the public internet 954. Some cloud services 956 may be isolated from other cloud services 956, and control plane VCN 916 may be isolated from cloud services 956 that may not be in the same region as control plane VCN 916. For example, control plane VCN 916 may be located in “Region 1” and cloud service “Deployment 8” may be located in Region 1 and Region 2. If a call to deployment 8 is made by a service gateway 936 included in a control plane VCN 916 in region 1, the call may be sent to deployment 8 in region 1. In this example, control plane VCN 916, and thus deployment 8 in region 1, may not be communicatively coupled to or in communication with deployment 8 in region 2.

[0112] 10 is a block diagram 1000 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1002 (e.g., service operator 802 of FIG. 8 ) can be communicatively coupled to a secure host tenant 1004 (e.g., secure host tenant 804 of FIG. 8 ), which can include a virtual cloud network (VCN) 1006 (e.g., VCN 806 of FIG. 8 ) and a secure host subnet 1008 (e.g., secure host subnet 808 of FIG. 8 ). The VCN 1006 can include an LPG 1010 (e.g., LPG 810 of FIG. 8 ), which can be communicatively coupled to an SSH VCN 1012 (e.g., SSH VCN 812 of FIG. 8 ) via an LPG 1010 included in the SSH VCN 1012. The SSH VCN 1012 can include an SSH subnet 1014 (e.g., SSH subnet 814 in FIG. 8 ), which can be communicatively coupled to a control plane VCN 1016 via an LPG 1010 included in the control plane VCN 1016 (e.g., control plane VCN 816 in FIG. 8 ) and to a data plane VCN 1018 via an LPG 1010 included in the data plane VCN 1018 (e.g., data plane 818 in FIG. 8 ). The control plane VCN 1016 and the data plane VCN 1018 can be included in a service tenant 1019 (e.g., service tenant 819 in FIG. 8 ).

[0113] The control plane VCN 1016 may include a control plane DMZ tier 1020 (e.g., control plane DMZ tier 820 of FIG. 8 ) that may include a load balancer (LB) subnet 1022 (e.g., LB subnet 822 of FIG. 8 ), a control plane app tier 1024 (e.g., control plane app tier 824 of FIG. 8 ) that may include an app subnet 1026 (e.g., similar to app subnet 826 of FIG. 8 ), and a control plane data tier 1028 (e.g., control plane data tier 828 of FIG. 8 ) that may include a DB subnet 1030. The LB subnet 1022 included in the control plane DMZ layer 1020 can be communicatively coupled to an app subnet 1026 included in the control plane app layer 1024 and can be communicatively coupled to an Internet gateway 1034 (e.g., Internet gateway 834 of FIG. 8 ) that can be included in the control plane VCN 1016, and the app subnet 1026 can be communicatively coupled to a DB subnet 1030 included in the control plane data layer 1028 and can be communicatively coupled to a service gateway 1036 (e.g., service gateway of FIG. 8 ) and a network address translation (NAT) gateway 1038 (e.g., NAT gateway 838 of FIG. 8 ). The control plane VCN 1016 can include the service gateway 1036 and the NAT gateway 1038.

[0114] The data plane VCN 1018 may include a data plane app layer 1046 (e.g., data plane app layer 846 of FIG. 8 ), a data plane DMZ layer 1048 (e.g., data plane DMZ layer 848 of FIG. 8 ), and a data plane data layer 1050 (e.g., data plane data layer 850 of FIG. 8 ). The data plane DMZ layer 1048 may include a LB subnet 1022, which may be communicatively coupled to a trusted app subnet 1060, which may be communicatively coupled to an untrusted app subnet 1062 of the data plane app layer 1046 and an Internet gateway 1034 included in the data plane VCN 1018. The trusted app subnet 1060 may be communicatively coupled to a service gateway 1036 included in the data plane VCN 1018, which may be communicatively coupled to a NAT gateway 1038 included in the data plane VCN 1018, and a DB subnet 1030 included in the data plane data layer 1050. The untrusted app subnet 1062 can be communicatively connected to a service gateway 1036 included in the data plane VCN 1018 and a DB subnet 1030 included in the data plane data layer 1050. The data plane data layer 1050 can include a DB subnet 1030 that can be communicatively coupled to a service gateway 1036 included in the data plane VCN 1018.

[0115] The untrusted app subnet 1062 may include one or more primary VNICs 1064(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1066(1)-(N). Each tenant VM 1066(1)-(N) may be communicatively coupled to a respective app subnet 1067(1)-(N) that may be included in a respective container egress VCN 1068(1)-(N) that may be included in a respective customer tenant 1070(1)-(N). Each secondary VNIC 1072(1)-(N) may facilitate communication between the untrusted app subnet 1062 included in the data plane VCN 1018 and the app subnet included in the container egress VCN 1068(1)-(N). Each container egress VCN 1068(1)-(N) may include a NAT gateway 1038 that may be communicatively coupled to the public Internet 1054 (e.g., public Internet 854 of FIG. 8).

[0116] An Internet gateway 1034 included in the control plane VCN 1016 and included in the data plane VCN 1018 can be communicatively coupled to a metadata management service 1052 (e.g., metadata management system 852 of FIG. 8 ), which can be communicatively coupled to the public Internet 1054. The public Internet 1054 can be communicatively coupled to a NAT gateway 1038 included in the control plane VCN 1016 and included in the data plane VCN 1018. A service gateway 1036 included in the control plane VCN 1016 and included in the data plane VCN 1018 can be communicatively coupled to cloud services 1056.

[0117] In some embodiments, the data plane VCN 1018 can be integrated with a customer tenant 1070. This integration can be beneficial or desirable for an IaaS provider's customer, such as when they want support in running their code. A customer may provide code to run that may be disruptive, communicate with other customer resources, or cause other undesirable effects. In response, the IaaS provider can decide whether to run code provided to the IaaS provider by the customer.

[0118] In some examples, a customer of an IaaS provider may grant the IaaS provider temporary network access and request functionality to be added to the data plane layer app 1046. The code to perform the functionality may be executed in the VMs 1066(1)-(N), and the code may not be configured to run elsewhere on the data plane VCN 1018. Each VM 1066(1)-(N) may be connected to one customer tenant 1070. Each container 1071(1)-(N) contained in the VMs 1066(1)-(N) may be configured to execute code. In this case, double isolation may exist (e.g., code execution in container 1071(1)-(N), container 1071(1)-(N) may be contained in at least VMs 1066(1)-(N) contained in the untrusted app subnet 1062), which may help prevent erroneous or unwanted code from damaging the IaaS provider's network or damaging another customer's network. The containers 1071(1)-(N) may be communicatively coupled to the customer tenants 1070 and may be configured to send or receive data from the customer tenants 1070. The containers 1071(1)-(N) may not be configured to send or receive data from any other entities in the data plane VCN 1018. Once the code execution is complete, the IaaS provider may kill or otherwise discard the containers 1071(1)-(N).

[0119] In some embodiments, the trusted app subnet 1060 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted app subnet 1060 may be communicatively coupled to the DB subnet 1030 and configured to perform CRUD operations on the DB subnet 1030. The untrusted app subnet 1062 may be communicatively coupled to the DB subnet 1030, but in this embodiment, the untrusted app subnet may be configured to perform read operations on the DB subnet 1030. Containers 1071(1)-(N) that may be included in each customer's VMs 1066(1)-(N) and that may execute code from the customer may not be communicatively coupled to the DB subnet 1030.

[0120] In other embodiments, the control plane VCN 1016 and the data plane VCN 1018 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 1016 and the data plane VCN 1018. However, communication may occur indirectly through at least one method. The LPG 1010 may be established by an IaaS provider that may facilitate communication between the control plane VCN 1016 and the data plane VCN 1018. In another example, the control plane VCN 1016 or the data plane VCN 1018 may make a call to a cloud service 1056 through a service gateway 1036. For example, a call from the control plane VCN 1016 to the cloud service 1056 may include a request for a service that may communicate with the data plane VCN 1018.

[0121] 11 is a block diagram 1100 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1102 (e.g., service operator 802 of FIG. 8 ) can be communicatively coupled to a secure host tenant 1104 (e.g., secure host tenant 804 of FIG. 8 ) which can include a virtual cloud network (VCN) 1106 (e.g., VCN 806 of FIG. 8 ) and a secure host subnet 1108 (e.g., secure host subnet 808 of FIG. 8 ). The VCN 1106 can include an LPG 1110 (e.g., LPG 810 of FIG. 8 ) which can be communicatively coupled to an SSH VCN 1112 (e.g., SSH VCN 812 of FIG. 8 ) via an LPG 1110 included in the SSH VCN 1112. SSH VCN 1112 can include an SSH subnet 1114 (e.g., SSH subnet 814 in FIG. 8 ), which can be communicatively coupled to a control plane VCN 1116 (e.g., control plane VCN 816 in FIG. 8 ) via an LPG 1110 included in the control plane VCN 1116 and to a data plane VCN 1118 (e.g., data plane 818 in FIG. 8 ) via an LPG 1110 included in the control plane VCN 1116. The control plane VCN 1116 and the data plane VCN 1118 can be included in a service tenant 1119 (e.g., service tenant 819 in FIG. 8 ).

[0122] The control plane VCN 1116 may include a control plane DMZ layer 1120 (e.g., control plane DMZ layer 820 of FIG. 8 ) which may include a LB subnet 1122 (e.g., LB subnet 822 of FIG. 8 ), a control plane app layer 1124 (e.g., control plane app layer 824 of FIG. 8 ) which may include an app subnet 1126 (e.g., app subnet 826 of FIG. 8 ), and a control plane data layer 1128 (e.g., control plane data layer 828 of FIG. 8 ) which may include a DB subnet 1130 (e.g., DB subnet 1030 of FIG. 10 ). The LB subnet 1122 included in the control plane DMZ tier 1120 may be communicatively coupled to an app subnet 1126 included in the control plane app tier 1124 and to an Internet gateway 1134 (e.g., Internet gateway 834 in FIG. 8 ) that may be included in the control plane VCN 1116, and the app subnet 1126 can be communicatively coupled to a DB subnet 1130 and a service gateway 1136 (e.g., service gateway in FIG. 8 ) and a network address translation (NAT) gateway 1138 (e.g., NAT gateway 838 in FIG. 8 ) included in the control plane data tier 1128. The control plane VCN 1116 may include the service gateway 1136 and the NAT gateway 1138.

[0123] The data plane VCN 1118 may include a data plane app layer 1146 (e.g., data plane app layer 846 in FIG. 8 ), a data plane DMZ layer 1148 (e.g., data plane DMZ layer 848 in FIG. 8 ), and a data plane data layer 1150 (e.g., data plane data layer 850 in FIG. 8 ). The data plane DMZ layer 1148 may include a LB subnet 1122, which may be communicatively coupled to a trusted app subnet 1160 (e.g., trusted app subnet 1060 in FIG. 10 ) and an untrusted app subnet 1162 (e.g., untrusted app subnet 1062 in FIG. 10 ) of the data plane app layer 1146 and the Internet gateway 1134 included in the data plane VCN 1118. The trusted app subnet 1160 may be communicatively coupled to a service gateway 1136 included in the data plane VCN 1118, a NAT gateway 1138 included in the data plane VCN 1118, and a DB subnet 1130 included in the data plane data layer 1150. The untrusted app subnet 1162 can be communicatively coupled to a service gateway 1136 included in the data plane VCN 1118 and a DB subnet 1130 included in the data plane data layer 1150. The data plane data layer 1150 can include a DB subnet 1130 that can be communicatively coupled to a service gateway 1136 included in the data plane VCN 1118.

[0124] The untrusted app subnet 1162 may include primary VNICs 1164(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1166(1)-(N) that reside within the untrusted app subnet 1162. Each tenant VM 1166(1)-(N) may execute code within a respective container 1167(1)-(N) and may be communicatively coupled to an app subnet 1126 that may be included in a data plane app layer 1146 that may be included in a container egress VCN 1168. Each secondary VNIC 1172(1)-(N) may facilitate communication between the untrusted app subnet 1162 included in the data plane VCN 1118 and the app subnet included in the container egress VCN 1168. The container egress VCN may include a NAT gateway 1138 that may be communicatively coupled to the public Internet 1154 (e.g., public Internet 854 of FIG. 8 ).

[0125] The Internet gateway 1134 included in the control plane VCN 1116 and included in the data plane VCN 1118 can be communicatively coupled to a metadata management service 1152 (e.g., metadata management system 852 of FIG. 8 ), which can be communicatively coupled to the public Internet 1154. The public Internet 1154 can be communicatively coupled to a NAT gateway 1138 included in the control plane VCN 1116 and included in the data plane VCN 1118. The service gateway 1136 included in the control plane VCN 1116 and included in the data plane VCN 1118 can be communicatively coupled to cloud services 1156.

[0126] In some examples, the pattern illustrated by the architecture of block diagram 1100 of FIG. 11 may be an exception to the pattern illustrated by the architecture of block diagram 1000 of FIG. 10 and may be desirable for an IaaS provider's customer when the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected area). Each container 1167(1)-(N) contained in each customer's VM 1166(1)-(N) is accessible in real time by the customer. The containers 1167(1)-(N) may be configured to make calls to a respective secondary VNIC 1172(1)-(N) contained in the app subnet 1126 of the data plane app tier 1146 that may be contained in a container egress VCN 1168. The secondary VNIC 1172(1)-(N) may send the call to a NAT gateway 1138 that may send the call to the public Internet 1154. In this example, containers 1167(1)-(N) that a customer can access in real time can be isolated from control plane VCN 1116 and can be isolated from other entities included in data plane VCN 1118. Containers 1167(1)-(N) may be isolated from resources from other customers.

[0127] In another example, a customer can use container 1167(1)-(N) to invoke cloud service 1156. In this example, the customer can execute code within container 1167(1)-(N) that requests a service from cloud service 1156. Container 1167(1)-(N) can send the request to secondary VNIC 1172(1)-(N), which can send the request to a NAT gateway that can send the request to public Internet 1154. Public Internet 1154 can send the request via Internet gateway 1134 to LB subnet 1122 included in control plane VCN 1116. In response to determining that the request is valid, the LB subnet can send the request to app subnet 1126, which can send the request to cloud service 1156 via service gateway 1136.

[0128] It should be understood that the IaaS architectures 800, 900, 1000, 1100 depicted in the figures may have components other than those depicted. Additionally, the embodiments depicted in the figures are only some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS systems may have more or fewer components than depicted, may combine two or more components, or may have a different configuration or arrangement of components.

[0129] In certain embodiments, the IaaS system described herein may include a suite of application, middleware, and database services products that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such an IaaS system is Oracle Cloud Infrastructure (OCI), offered by the present assignee.

[0130] 12 illustrates an exemplary computer system 1200 upon which various embodiments may be implemented. System 1200 may be used to implement any of the computer systems described above. As shown, computer system 1200 includes a processing unit 1204 that communicates with a number of peripheral subsystems via a bus subsystem 1202. These peripheral subsystems may include a processing accelerator 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 includes a tangible computer-readable storage medium 1222 and a system memory 1210.

[0131] Bus subsystem 1202 provides a mechanism that allows the various components and subsystems of computer system 1200 to communicate with each other as intended. Although bus subsystem 1202 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. It may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0132] The processing unit 1204 may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers) and controls the operation of the computer system 1200. One or more processors may be included in the processing unit 1204. These processors may include single-core processors or multi-core processors. In particular embodiments, the processing unit 1204 may be implemented as one or more independent processing units 1232 and / or 1234 with a single-core processor or a multi-core processor included in each processing unit. In other embodiments, the processing unit 1204 may be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0133] In various embodiments, the processing unit 1204 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any time, some or all of the program code being executed may reside in the processor 1204 and / or in the memory subsystem 1218. Through appropriate programming, the processor 1204 may provide the various functions discussed above. The computer system 1200 may further include a processing accelerator 1206, which may include a digital signal processor (DSP), a special purpose processor, or the like.

[0134] The I / O subsystem 1208 can include user interface input devices and user interface output devices. User interface input devices can include keyboards, pointing devices such as mice or trackballs, touch pads or touch screens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices can include, for example, motion sensing and / or gesture recognizers such as a Microsoft Kinect® motion sensor, allowing a user to control and interact with input devices such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and voice commands. User interface input devices can also include eye gesture recognizers such as a Google Glass® blink detector that detects eye activity from a user (e.g., "blinking" while taking a picture and / or selecting a menu) and translates eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., the Siri® navigator) through voice commands.

[0135] User interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound machines. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0136] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. For example, user interface output devices include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0137] Computer system 1200 may include a storage subsystem 1218 that comprises software elements shown as currently located in system memory 1210. System memory 1210 may store program instructions that are loadable and executable on processing unit 1204, as well as data generated during the execution of these programs.

[0138] Depending on the configuration and type of computer system 1200, the system memory 1210 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 1204. In some implementations, the system memory 1210 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in ROM. By way of example and not limitation, the system memory 1210 also illustrates application programs 1212, program data 1214, and an operating system 1216, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like. By way of example, operating systems 1216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 12 OS, and Palm® OS operating systems.

[0139] The storage subsystem 1218 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by the processor, provide the functionality described above may be stored in the storage subsystem 1218. These software modules or instructions may be executed by the processing unit 1204. The storage subsystem 1218 may also provide a repository for storing data used in accordance with the present disclosure.

[0140] Storage subsystem 1200 may also include a computer readable storage medium reader 1220 that can be further connected to a computer readable storage medium 1222. Together, and optionally in combination with system memory 1210, computer readable storage medium 1222 can comprehensively represent remote, local, fixed, and / or removable storage devices, as well as storage media for containing, storing, transmitting, and retrieving computer readable information on a temporary and / or more permanent basis.

[0141] The computer readable storage medium 1222 containing the code or portions of code may include any suitable medium known or used in the art, including, but not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable medium. This may also include intangible computer readable media, such as a data signal, data transmission, or any other medium that can be used to transmit the desired information and that can be accessed by the computing system 1200.

[0142] As an example, the computer readable storage medium 1222 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads or writes to a removable non-volatile optical disk, such as a CD-ROM, DVD, Blu-Ray® disk, or other optical medium. The computer readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage medium 1222 may also include a solid state drive (SSD) based on non-volatile memory, such as a flash memory-based SSD, an enterprise flash drive, an SSD based on volatile memory, such as solid state RAM, dynamic RAM, static RAM, such as solid state ROM, a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD that uses a combination of DRAM and a flash memory-based SSD. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 1200.

[0143] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 serves as an interface for transmitting and receiving data from the computer system 1200 to and from other systems. For example, the communication subsystem 1224 may enable the computer system 1200 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1224 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution)), WiFi (IEEE 802.11 family standard, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communication subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0144] In some embodiments, the communications subsystem 1224 may also receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., on behalf of one or more users who may use the computer system 1200.

[0145] As an example, the communications subsystem 1224 may be configured to receive data feeds 1226 in real time from users of social networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party information sources.

[0146] Additionally, the communications subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or essentially unlimited with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0147] The communications subsystem 1224 may also be configured to output structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1200.

[0148] The computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or other data processing system.

[0149] Due to the ever-changing nature of computers and networks, the description of the computer system 1200 shown in the figure is intended as a specific example only. Many other configurations are possible, with more or fewer components than the system shown in the figure. For example, customized hardware may also be used, or particular elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may be used. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other approaches and / or methods for implementing various embodiments.

[0150] Although specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are within the scope of the disclosure. The embodiments are not limited to operating in any particular data processing environment, but may freely operate in multiple data processing environments. Furthermore, while the embodiments have been described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that the scope of the disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the above-described embodiments may be used individually or in combination.

[0151] Furthermore, while embodiments have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. The embodiments may be implemented using only hardware, only software, or a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors. Thus, where a component or module is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, with different pairs of processes using different techniques and the same pair of processes using different techniques at different times.

[0152] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made without departing from the broader spirit and scope of the appended claims. Accordingly, although certain disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be within the scope of the following claims.

[0153] Use of the terms "a," "an," "the," and similar referents in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprise," "have," "include," and "contain" should be construed as open-ended terms (i.e., meaning "including but not limited to"), unless otherwise noted. The term "connected" should be construed as being partially or wholly contained within, attached to, or connected to, even if there is something intervening. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each individual value falling within the range, unless otherwise stated herein, and each individual value is incorporated into the specification as if it were individually set forth herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by context. Any examples provided herein, or the use of exemplary language (e.g., "such as"), are intended merely to better illustrate the embodiments and do not impose limitations on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0154] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as generally used to indicate that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless specifically stated otherwise. Thus, such disjunctive language is generally not intended to, and should not, imply that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0155] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the present disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to adopt such variations as necessary, and the present disclosure may be carried out in ways other than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, unless otherwise indicated herein, this disclosure encompasses any combination of the above-described elements in all possible variations thereof.

[0156] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0157] In the foregoing specification, aspects of the disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure can be used individually or in combination. Moreover, the embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive. Various modifications and equivalents include relevant suitable combinations of the features disclosed in the embodiments.

Claims

1. 1. A method comprising: a distributed computing system implementing an initial data store configured to store data associated with a user of the initial data store, the initial data store having an initial store state and designated as a primary data store; The method comprises: the distributed computing system implementing a target data store having a target store state; the distributed computing system migrating a portion of the data associated with the user stored in the initial data store to the target data store; During said transfer of said portion of said data: receiving a request, associated with a key, to create a first data entity, at the distributed computing system; In response to the request, the distributed computing system determines that the initial data store is designated as the primary data store; and the distributed computing system determining the initial store state associated with the initial data store; In response to determining the initial store state, the distributed computing system validates that the key does not exist in the target data store; and The method, further comprising: the distributed computing system creating the first data entity in the initial data store, wherein creating the first data entity comprises committing the first data entity to the initial data store.

2. the distributed computing system designating the target data store as the primary data store; receiving, by the distributed computing system, a second request to create a second data entity, the second request being associated with a second key; In response to the second request, the distributed computing system determines that the target data store is designated as the primary data store; and the distributed computing system determining the target store state associated with the target data store; In response to the determining of the target store state, the distributed computing system verifies that the second key does not exist in the initial data store; 2. The method of claim 1, further comprising: the distributed computing system creating the second data entity in the target data store, and creating the second data entity comprises committing the second data entity to the target data store.

3. the second data entity includes new immutable data in the target data store; the distributed computing system designating the initial data store as the primary data store; the distributed computing system transferring an additional portion of the data stored in the target data store to the initial data store; During said transfer of said additional portion of said data: receiving, by the distributed computing system, a third request to create a third data entity, the third request being associated with a third key; in response to the third request, the distributed computing system determines that the initial data store is designated as the primary data store; and the distributed computing system determining the initial store state associated with the initial data store; In response to the determining of the initial store state, the distributed computing system verifies that the third key does not exist in the target data store; and 3. The method of claim 2, further comprising: the distributed computing system creating the third data entity in the initial data store, and creating the third data entity comprises committing the third data entity to the initial data store.

4. The method of claim 1 , wherein the initial store state can be open or closed, and the target store state can be open or closed.

5. The method of claim 1 , wherein the data includes immutable data, ephemeral data, and key data.

6. The method of claim 5 , wherein migrating the portion of the data comprises replicating the immutable data from the initial data store to the target data store.

7. prior to committing the first data entity to the initial data store, the distributed computing system determining the initial store state; The method of claim 1 , further comprising the distributed computing system committing the first data entity to the initial data store.

8. one or more processors; one or more memories that store computer-executable instructions, The computer-executable instructions, when executed by the one or more processors, cause the distributed computing system to perform a method according to any one of claims 1 to 7.

9. A program that causes a distributed computing system to execute a method described in any one of claims 1 to 7.

Images