A mechanism for a network virtualization device to provide customer VCN network encryption using customer-managed keys

JP2024502770A5Active Publication Date: 2024-12-04ORACLE INT CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023538778
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-23
Filing Date
2021-12-16
Publication Date
2024-12-04
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

Existing virtual cloud network (VCN) encryption systems often require hosts to manage encryption keys, posing a security risk as hosts may have access to sensitive customer data, and existing encryption algorithms like RC4 and 3DES face performance penalties or security vulnerabilities.

Method used

Implementing network interface cards (NICs) with cryptographic processors and memory devices to enable customer-managed encryption keys, using key management services to securely distribute keys without exposing them to hosts, and ensuring each virtual machine has dedicated cryptographic resources for secure encryption and decryption.

Benefits of technology

Ensures secure encryption of VCN data with customer-controlled keys, preventing host access and minimizing performance impact, thus enhancing data security and reducing vulnerabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To allow a customer to control network keys in a virtual cloud network, a network interface card such as a smart NIC is used to provide encryption such as a network encryption virtual function (NEVF) for virtual machines. The NEVF includes a memory device (e.g., SRAM) and a crypto processor (e.g., a crypto core). The memory device stores an encryption key. The crypto processor encrypts data to and from the virtual machines in the virtual cloud network using the encryption key. A key management service can be used to securely transfer the encryption key to the NEVF. Having one NEVF per virtual machine can allow a customer to manage encryption keys for the virtual cloud network.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Application No. 17 / 133,526, filed December 23, 2020, and entitled "MECHANISM TO PROVIDE CUSTOMER VCN NETWORK ENCRYPTION USING CUSTOMER-MANAGED KEYS IN NETWORK VIRTUALIZATION DEVICE," the entire contents of which are incorporated herein by reference.

[0002] This application is related to U.S. Application No. 17 / 133,523, filed on December 23, 2020, and entitled “END-TO-END NETWORK ENCRYPTION FROM CUSTOMER ON-PREMISE NETWORK TO CUSTOMER VIRTUAL CLOUD NETWORK USING CUSTOMER-MANAGED KEYS,” the entire contents of which are incorporated herein by reference. [Background technology]

[0003] background A Virtual Cloud Network (VCN) is a customizable private network. Similar to a traditional data center network, a VCN controls the network environment. This includes allocating private IP addresses, creating subnets, creating route tables, and configuring firewalls. A tenant can have multiple VCNs, allowing for grouping and isolation of related resources. Summary of the Invention [Problem to be solved by the invention]

[0004] Data in a VCN can be encrypted for security. One goal of encryption is to transform plaintext data into unintelligible ciphertext based on a key such that it is very difficult (e.g., computationally infeasible) to convert the ciphertext back into the corresponding plaintext without knowledge of the correct key. In a symmetric cryptosystem, the same key is used to both encrypt and decrypt the same data. In some encryption algorithms, the key length can be 128 bits, 192 bits, or 256 bits. In some encryption standard (e.g., Data Encryption Standard (DES)) algorithms, message data is encrypted by passing it through the DES algorithm three times. 3DES can provide a high degree of message security, but it comes with a performance penalty. The degree of the performance penalty can vary depending on the speed of the processor performing the encryption. The RC4 algorithm, developed by RSA Data Security, Inc., has become the international standard for high-speed data encryption. RC4 is a variable key-length stream cipher that operates several times faster than DES, so it can encrypt large data transfers with minimal impact on performance. Network data encryption provides data privacy so that unauthorized parties cannot view plaintext data as it travels across the network.

[0005] Quick Overview The present disclosure relates generally to virtual cloud networks (VCNs). More specifically, but without limitation, techniques are described for encrypting data in a VCN that allows customers to maintain and manage one or more encryption keys without the VCN host having access to the encryption keys. The lack of host access to the encryption keys can increase the level of data security in the VCN. Various embodiments are described herein, including methods, systems, non-transitory computer readable storage media storing programs, code, or instructions executable by one or more processors, and the like. [Means for solving the problem]

[0006] In certain embodiments, a network interface card (e.g., a smart NIC) is used to provide encryption such as a network encryption virtual function (NEVF) for virtual machines so that a customer can control the network encryption keys of the virtual cloud network. The NEVF includes a memory device (e.g., SRAM) and a crypto processor (e.g., a crypto core). The memory device stores the encryption key. The crypto processor uses the encryption key to encrypt data to and from the virtual machines in the virtual cloud network. A key management service is used to securely transfer the encryption key to the NEVF. Having one NEVF per virtual machine can enable a customer to manage the encryption keys of the virtual cloud network. The administrator of the device (e.g., the host of the server with the network interface card running the NEVF) does not have access to the encryption keys.

[0007] In a particular embodiment, a system includes a first virtualization engine and a second virtualization engine. The first virtualization engine is configured to instantiate a first virtual machine. The first virtualization engine includes a first memory device and a first cryptographic processor. The first memory device is configured to store a first cryptographic key. The first cryptographic processor is configured to encrypt and decrypt data to and from the first virtual machine using the first cryptographic key. The second virtualization engine is configured to instantiate a second virtual machine. The second virtualization engine includes a second memory device and a second cryptographic processor. The second memory device is configured to store a second cryptographic key. The second cryptographic processor is configured to encrypt and decrypt data to and from the second virtual machine using the second cryptographic key. The first virtualization engine and the second virtualization engine are part of a same device (e.g., a network interface card). An administrator of the device does not have access to the first key or the second key.

[0008] In a particular embodiment, the first virtualization engine is configured to receive the first key from a key management service and / or store the first key in the first memory device, the key management service providing the first key to the device without an administrator of the device having access to the first key for the customer, the first memory device is a static random access memory device including a capacitor and a transistor, the first virtualization engine and the second virtualization engine are each configured to instantiate only one virtual machine at a time, the first virtual machine is part of a first virtual cloud network and the second virtual machine is part of a second virtual cloud network, and the first cryptographic processor is configured to instantiate the first virtual machine. and a key management service configured to provide inline data encryption for one virtual machine, the first virtualization engine configured to provide network routing of data packets for the first virtualization machine, the first encryption key is the same as the second encryption key, the device is a network interface card, the first key and / or the second key are periodically updated, the first virtualization engine and the second virtualization engine provide credentials to the key management service to receive the first key, the first virtual machine and the second virtual machine are part of a subnet, the first virtual machine and the second virtual machine are part of different virtual cloud networks, and the first key and the second key are refreshed simultaneously.

[0009] In yet another embodiment, a method includes instantiating a first virtual machine using a first virtualization engine, instantiating a second virtual machine using a second virtualization engine, storing a first key in a first memory device of the first virtualization engine, storing a second key in a second memory device of the second virtualization engine, and encrypting data of the first virtual machine using a first cryptographic processor, the first cryptographic processor being part of the first virtualization engine, and / or encrypting data of the second virtualization engine using a second cryptographic processor, the second cryptographic processor being part of the second virtualization engine. The first virtualization engine and the second virtualization engine are part of a same device. An administrator of the device does not have access to the first key or the second key.

[0010] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram of an embodiment of a system of network virtualization devices. [Diagram 2] FIG. 1 illustrates one embodiment of multiple network virtualization devices that receive encryption keys from a key management service. [Diagram 3] FIG. 1 illustrates an embodiment of a network virtualization device that supports multiple customers with different encryption keys. [Figure 4] FIG. 1 is a flow chart diagram illustrating an embodiment of a process for encrypting data in a virtual cloud network. [Diagram 5] FIG. 2 is a flow chart diagram of one embodiment of a process for data encryption for multiple virtual cloud networks using multiple cryptographic keys. [Figure 6]FIG. 2 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 7] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 8] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 9] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an exemplary computer system according to at least one embodiment. [Figure 11] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual cloud network or overlay cloud network hosted by a cloud service provider infrastructure according to certain embodiments. [Figure 12] FIG. 2 is a simplified architectural diagram of physical components in a physical network within a CSPI, according to certain embodiments. [Figure 13] FIG. 2 illustrates an exemplary arrangement within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to certain embodiments. [Figure 14] FIG. 1 illustrates connectivity between a host machine and an NVD to provide I / O virtualization to support multitenancy, according to a particular embodiment. [Figure 15] FIG. 2 is a simplified block diagram of a physical network provided by the CSPI, according to certain embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Detailed Description In the accompanying drawings, similar components and / or features may have the same reference level. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used in this specification, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label.

[0013] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily intended to be construed as preferred or advantageous over other embodiments or designs.

[0014] A Virtual Cloud Network (VCN) is a customizable private network. A host may provide computing hardware and / or software for a customer to set up a VCN. Typically, the host manages the encryption, if any, of the VCN. A customer may not want the host to manage the encryption, fearing that a data leak on the host could compromise customer data or concern how much customer data the host can access. This description relates to a mechanism to provide VCN network encryption using customer-managed keys. The customer-managed keys can be delivered on a smart NIC. A smart NIC is a network interface card (e.g., a network adapter) that offloads processing tasks that would normally be handled by a CPU. A smart NIC can perform functions such as encryption, decryption, routing, and firewalls.

[0015] The smart NIC can be used to support a Network Encryption Virtual Function (NEVF) with a dedicated crypto accelerator and / or SRAM. The crypto accelerator can be used for inline packet encryption. The SRAM can be used to store encryption keys. The NEVF assigned to a customer virtual machine (the customer virtual machine is part of the customer VCN) can be used for virtual machine network traffic encryption and to store the encryption keys. The hypervisor can map the customer virtual machine to the NEVF. The encryption keys can be securely provisioned in the NEVF within the customer VCN instance (e.g., virtual machine (VM) and / or bare metal (BM)).

[0016] Traffic exchanged between customer VCN instances is encrypted with encryption keys provisioned in the NEVF. Each instance in the VCN shares the same encryption key (e.g., opportunistic encryption). The control plane can provision and / or manage network encryption keys for customer instances (VMs and / or BMs) in the VCN. The control plane can be used to authenticate customer virtual functions and provision encryption keys from the key management service. Using certificates, the key management service and the NIC can securely share customer encryption keys. Thus, the customer can manage the network encryption keys (e.g., using application program interfaces and / or operating system commands). In some configurations, unique keys are shared between VM hosts in a VCN. The NEVF of each VM host stores the key and / or metadata associated with which key to use.

[0017] Referring to FIG. 1, a schematic diagram of one embodiment of a system for encrypting data of a virtual cloud network is shown. The network virtualization device 100 includes a first virtualization engine 104-1 and a second virtualization engine 104-2. The virtualization engine 104 instantiates and provides a virtual network interface card (VNIC) for each bare metal or virtual machine. The virtualization engine 104 may be a network encryption virtual function (NEVF). The network virtualization device 100 may be a physical card (e.g., a smart NIC) with dedicated resources (e.g., memory and / or processor) attached. In some embodiments, the network virtualization device 100 is a SmartTOR, a network appliance, or a service host.

[0018] Routing and / or forwarding of packets within the cloud service provider infrastructure may be performed by a network virtualization device. A network virtualization device may realize one or more virtual network functions, such as a virtual network interface card (VNIC), a virtual router, a virtual network gateway, and a network encryption virtual function. A network virtualization device is a virtual object realized using software (e.g., code or instructions executed by a processor). A network virtualization device is a hardware component that executes a virtual function, such as a virtual router (e.g., a virtual router device is a physical device that executes code that realizes a virtual router). For example, a VNIC is executed by a smart NIC, and the smart NIC is a network virtualization device. When a VNIC is executed by a host machine, the host machine is a network virtualization device. In some embodiments, a virtual router is executed by a top-of-rack (TOR) switch (the TOR is a virtual router device). A smart NIC is just one example of a network virtualization device.

[0019] A virtual machine 108 is a program on a computer that behaves like a separate computer inside the computer. Virtual machines can be created using virtualization software. Virtual machines 108 can perform a variety of virtual functions.

[0020] The virtualization engine 104 includes a memory device 112 and a cryptographic processor 116. The memory device 112 is configured to store keys, such as cryptographic keys. In some embodiments, the memory device 112 is a static random access memory (SRAM) device that includes capacitors and transistors.

[0021] The crypto processor 116 is configured to encrypt and / or decrypt data to and from the virtual machine 108 using a key stored in the memory device 112. An outgoing packet TX is a data packet sent from the virtual machine 108. An incoming packet RX is a data packet sent to the virtual machine 108. The outgoing packet TX is encrypted by the crypto processor 116. The incoming packet RX is decrypted by the crypto processor 116. The outgoing packet TX and / or the incoming packet RX are encrypted / decrypted using in-line encryption / decryption. In some embodiments, the virtualization engine 104 is configured to provide network routing of data packets (e.g., the virtualization engine 104 provides routing data for the outgoing packet TX).

[0022] In the illustrated embodiment, the virtualization engine 104 is configured to instantiate only one virtual machine 108 at a time. This can increase security by allowing dedicated cryptographic resources (e.g., memory device 112 and / or crypto processor 116) for each virtual machine 108.

[0023] In the illustrated embodiment, the first virtualization engine 104-1 includes a first memory device 112-1 and a first crypto processor 116-1, and the second virtualization engine 104-2 includes a second memory device 112-2 and a second crypto processor 116-2. Thus, the first virtualization engine 104-1 and the second virtualization engine 104-2 are part of the same device. Although two virtualization engines 104 are shown as being part of the network virtualization device 100, it should be understood that more than two virtualization engines 104 may be part of the network virtualization device 100 (e.g., three, five, ten, one hundred, or more virtualization engines 104 may be part of the network virtualization device 100).

[0024] In some embodiments, network virtualization device 100 is used to realize virtual functions such as VNICs, virtual routers, and / or NEVFs. Network virtualization device 100 can be a card (e.g., smart NIC) in a server. Network virtualization device 100 is managed (e.g., owned and / or operated) by a host for one or more customers to set up one or more VCNs. The host is the administrator of network virtualization device 100. The key can be stored (e.g., encrypted) in memory device 112 such that the host (e.g., administrator) does not have access to the key stored in memory device 112. The host's lack of access to the key can ensure data security for the customer even if the security of the host is compromised.

[0025] The Ethernet bridge 120 allows the first virtualization engine 104-1 to communicate over the Internet and / or with the second virtualization engine 104-2 (e.g., if the first virtual machine 108-1 and the second virtual machine 108-2 are part of the same virtual cloud network). The hypervisor 124 is used by the host to share the computing system's resources among the virtual machines 108.

[0026] FIG. 2 illustrates one embodiment of multiple virtualization engines 104 receiving encryption keys 204 from a key management service 208. In the embodiment shown, there are three virtualization engines 104. Although three virtualization engines 104 are shown, more or fewer virtualization engines 104 can be used. Each virtualization engine 104 instantiates one (or in some embodiments only one) virtual machine 108. The virtual machines 108 in FIG. 2 are part of a common virtual cloud network (VCN) 212. Because the virtualization engines 104 are part of the same VCN 212, each virtualization engine 104 receives the same key (e.g., the same encryption key 204), which allows the virtualization engines 104 to securely communicate with each other (e.g., using the Ethernet bridge 120 shown in FIG. 1) using encrypted data. Thus, a first virtualization engine 104-1 is configured to receive an encryption key 204 from the key management service 208. The first virtualization engine 104-1 stores the encryption key 204 in the first memory device 112-1.

[0027] The VCN control plane 216 is configured to manage, monitor, and / or modify cloud infrastructure resources. The key management service 208 can provide the encryption keys 204 to the virtualization engine 104 for the customer without the host of the network virtualization device 100 having access to the encryption keys 204. With the host (e.g., the cloud service provider) not having access to the encryption keys 204, the customer controls the security of the VCN 212.

[0028] To populate the key in the memory device 112 without the host having access to the key, the customer can initialize the encryption key 204 in a key management service 208 (e.g., OCI's Cloud Key Management Service) and / or grant the virtualization engine 104 permission to access the key. In the cloud network, each resource can be assigned an identity principal (e.g., OCI's Cloud Identity Principle), which can give each resource credentials to authenticate to other cloud resources. The virtualization engine 104 (e.g., NEVF) is a resource and can be assigned an identity principal. The virtualization engine 104 can be authenticated to the key management service 208. The virtualization engine 104 can request a customer key (e.g., encryption key 204).

[0029] The virtualization engine 104 can request an encryption key 204 from the key management service 208 (e.g., authenticate itself using an identity principal to request a key). The key management system 208 can push keys (and / or updates) to the virtualization engine 104 (e.g., after authenticating the virtualization engine 104). The request or push of keys occurs without the cloud network host provisioning the keys to the virtualization engine 104 (e.g., in the SRAM of the virtualization engine 104). Security protocols can be pre-shared between the key management system 208 and the virtualization engine 104, so that the key management system can route data to the virtualization engine 104 with the pre-shared secret to initialize the keys in the memory of the virtualization engine 104.

[0030] The same encryption key 204 is delivered to each virtualization engine 104 supporting virtual machines 108 in the virtual cloud network 212. The encryption key 204 can be delivered to the virtualization engine 104 when the virtual machine 108 is instantiated. For example, when a customer creates a VCN 212, the customer can be asked whether or not to encrypt the VCN 212 (e.g., the customer can check a box to encrypt the VCN 212). When an instance of a virtual machine 108 is launched, an underlying protocol passes from the key management service 208 to the virtualization engine 104 and delivers the encryption key 204 to the memory device 112 of the virtualization engine 104. The encryption key 204 can be updated periodically (e.g., hourly, daily, weekly, etc.) and / or by the customer (e.g., the customer selects or defines an update schedule and / or selects to update the encryption key 204 immediately). In some embodiments, a customer generates multiple keys, and when it is time to rotate to a new key (e.g., at time T each day), the encryption key 204 is updated (e.g., each virtualization engine 104 contacts the key management service 208 at time T). The new key is published in the memory device 112. Key synchronization (e.g., automatic synchronization) can be used for multiple hosts (e.g., a synchronization counter in each virtualization engine 104 may be used to publish the new key in the memory device 112). In some embodiments, the encryption key 204 is rotated on each virtualization engine 104 based on some encryption scheme (e.g., the encryption key 204 is refreshed). Thus, the encryption key 204 can be pushed and / or refreshed simultaneously to or on each virtualization engine 104 that is part of the VCN 212. The key management service 208 can have one encryption relationship with the VCN 212, so the key management service 208 does not need to maintain relationships with different devices.

[0031] By encrypting data where the customer controls the key, the customer can be assured that the data is encrypted on the wire to the host. In some embodiments, the customer can use a Transport Layer Security (TLS) tunnel to communicate with the host. However, TLS has bugs and can have security issues. By allowing the customer to control the key using the key management service 208, TLS bugs and security issues can be avoided or reduced. In some embodiments, a TLS tunnel or a custom security protocol based on a pre-shared key can be used for secure communication between the NEVF and the key management service.

[0032] Although this description shows encryption keys 204 being distributed and managed at the level of VCN 212, similar processes and techniques can be used at other levels, such as the subnet level. For example, encryption keys can be used at L2 or L3 of VCN 212.

[0033] 3 illustrates an embodiment of a network virtualization device 100 supporting multiple customers with different encryption keys 304. The network virtualization device 100 includes a first virtualization engine 104-1, a second virtualization engine 104-2, a third virtualization engine 104-3, and a fourth virtualization engine 104-4. The first virtualization engine 104-1 is configured to instantiate a first virtual machine 108-1. The second virtualization engine 104-2 is configured to instantiate a second virtual machine 108-2. The third virtualization engine 104-3 is configured to instantiate a third virtual machine 108-3. The fourth virtualization engine 104-4 is configured to instantiate a fourth virtual machine 108-4.

[0034] The first virtual machine 108-1 and the third virtual machine 108-3 are part of a first VCN 312-1. The second virtual machine 108-2 and the fourth virtual machine 108-4 are part of a second VCN 312-2. The first VCN 312-1 is for a first customer. The second VCN 312-2 is for a second customer, which is not the same as the first customer. Thus, multiple NEVFs on one device may be used to support virtual machines that belong to different virtual cloud networks.

[0035] The first virtualization engine 104-1 and the third virtualization engine 104-3 receive the first encryption key 304-1 from the first key management service 208-1. The second virtualization engine 104-2 and the fourth virtualization engine 104-4 receive the second encryption key 304-2 from the second key management service 208-2. The second encryption key 304-2 is different from the first encryption key 304-1. In some embodiments, the virtualization engine 104 of the second VCN 312-2 receives the second encryption key 304-2 from the same key management service as the virtualization engine 104 of the first VCN 312-1 (e.g., the virtualization engine 104 of the second VCN 312-2 receives the second encryption key 304-2 from the first key management service 208-1).

[0036] The first virtualization engine 104-1 stores the first encryption key 304-1 in the first memory device 112-1. The second virtualization engine 104-2 stores the second encryption key 304-2 in the second memory device 112-2. Similarly, the third virtualization engine 104-3 and the fourth virtualization engine 104-4 each store their respective encryption keys 304 in the memory device 112. The host of the network virtualization device 100 does not have access to the first encryption key 304-1 or the second encryption key 304-2. Having one virtualization engine 104 per virtual machine 108 (e.g., by having dedicated resources such as one memory device 112 per virtual machine 108) allows one network virtualization device 100 to securely support multiple VCNs 312 for multiple customers. Each customer manages its own encryption key 304 for the VCN 312, so that each customer's data can be managed securely. Furthermore, even if the data of the host is compromised, the data of the VCN 312 will not be compromised, providing further security reliability to customers.

[0037] 4 is a flow chart illustrating one embodiment of a process 400 for encrypting data in a virtual cloud network. The process 400 begins at step 402 with instantiating a first virtual machine. For example, as shown in FIG. 1, the first virtualization engine 104-1 instantiates the first virtual machine 108-1. At step 404, a second virtual machine is instantiated. For example, as shown in FIG. 1, the second virtualization engine 104-2 instantiates the second virtual machine 108-2. As shown in FIG. 1, the virtualization engine 104 includes a memory device 112 and a cryptographic processor 116.

[0038] The first key is stored in a first memory device (step 406). For example, as shown in FIG. 2, the encryption key 204 is stored in the first memory device 112-1 of the first virtualization engine 104-1. The first key may be retrieved from (e.g., requested from or pushed by) a key management service. In some embodiments, the virtual cloud network customer controls the first key such that the host of the first virtualization engine 104-1 does not have access to the key.

[0039] The second key is stored in a second memory device (step 408). For example, as shown in FIG. 2, the encryption key 204 is stored in the second memory device 112-2 of the second virtualization engine 104-2. The second key can be retrieved from (e.g., requested from or pushed by) a key management service. In some embodiments, the customer of the virtual cloud network controls the second key such that the host of the virtualization engine 104 does not have access to the second key. As shown in FIG. 2, the second key may be the same as the first key. As shown in FIG. 3, the second key may be different from the first key. For example, the second key may be different from the first key because the second virtual machine is part of a different virtual cloud network than the first virtual machine.

[0040] In step 410, the data of the first virtual machine is encrypted using the first key. For example, in FIG. 1, the first crypto processor 116-1 encrypts the transmission data TX from the first virtual machine 108-1. In step 412, the data of the second virtual machine is encrypted using the second key. For example, in FIG. 1, the second crypto processor 116-2 encrypts the transmission data TX from the second virtual machine 108-2. If the first virtual machine 108-1 and the second virtual machine 108-2 are part of the same virtual cloud network, the first virtual machine 108-1 can use the same key to securely transmit data to and / or receive data from the second virtual machine 108-2 (see, for example, FIG. 2). However, if the first key is different from the second key, as shown in FIG. 3, the first virtual machine 108-1 does not send encrypted data to and / or receive encrypted data from the second virtual machine 108-2 (unless there is some additional arrangement between the first VCN 312-1 and the second VCN 312-2 in FIG. 3).

[0041] The first virtualization engine 104-1 and the second virtualization engine 104-2 are on the same device (e.g., part of the same card in a server). Because the virtualization engine 104 is used to instantiate only one virtual machine and / or because the virtualization engine has one or more dedicated resources, data to and from each virtual machine can be securely encrypted and / or decrypted without the host of the network virtualization device having access to the first key or the second key.

[0042] FIG. 5 is a flow chart illustrating an embodiment of a process 500 for data encryption for multiple virtual cloud networks using multiple encryption keys. The process 500 begins with instantiating a first set of virtual machines for a first customer at step 502. A first set of virtualization engines is used to instantiate the first set of virtual machines. The first set of virtual machines is part of a first virtual cloud network. In this embodiment, a set includes two or more. For example, in FIG. 3, the first virtualization engine 104-1 and the third virtual machine 108-1 are instantiated using the first virtual machine 108-1 and the third virtual machine 108-3, and the first virtual machine 108-1 and the third virtual machine 108-3 are part of the first virtual cloud network 312-1.

[0043] A second set of virtual machines for the second customer is instantiated (step 504). The second set of virtual machines is instantiated using the second set of virtualization engines. The second set of virtual machines are part of a second virtual cloud network. For example, in FIG. 3, the second virtualization engine 104-2 and the fourth virtual machine 108-2 and the fourth virtual machine 108-4 are instantiated using the second virtualization engine 104-2 and the fourth virtual machine 108-4, and the second virtual machine 108-2 and the fourth virtual machine 108-4 are part of the second virtual cloud network 312-2.

[0044] In step 506, a first key, e.g., first encryption key 304-1 of FIG. 3, is received by a virtualization engine used to instantiate virtual machines in the first virtual cloud network. A second key, e.g., second encryption key 304-2 of FIG. 3, is received by a virtualization engine used to instantiate virtual machines in the second virtual cloud network (step 508). The first key can be received from a first key management service and the second key can be received from a second key management service different from the first key management service.

[0045] The first key is used to encrypt data for a first customer, step 510. The second key is used to encrypt data for a second customer, step 512. The host does not have access to the first key and / or the second key.

[0046] Infrastructure as a service (IaaS) is a particular type of cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider may host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may also offer various services (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.) that accompany those infrastructure components. Thus, these services may be policy-driven, so that an IaaS user may be able to implement policies that drive load balancing to maintain application availability and performance.

[0047] In some examples, an IaaS customer may access resources and services over a wide area network (WAN), such as the Internet, and can utilize the cloud provider's services to install the remaining elements of the application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then utilize the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, and more.

[0048] In most cases, cloud computing models use the participation of a cloud provider, which may or may not be a third-party service specialized in providing (e.g., offering, renting, selling) IaaS. Also, an entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0049] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed below the hypervisor layer (e.g., server, storage, network hardware, and virtualization) by the cloud provider. Thus, the customer may be responsible for handling (e.g., on self-service virtual machines (e.g., that may be spun up on demand), middleware, and / or application deployment, etc.

[0050] In some instances, IaaS provisioning may refer to obtaining the computers or virtual hosts to be used and even installing the required libraries or services on them. In most cases, deployment does not include provisioning, which may need to be performed first.

[0051] In some cases, there are two distinct challenges in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything works. Second, there is the challenge of evolving the existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they each work together) can be described declaratively. In some examples, once the topology is defined, workflows can be generated that create and / or manage the different components described in the configuration files.

[0052] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shared computing resources), also known as a core network. In some examples, there may also be one or more security group rules and one or more virtual machines (VMs) provisioned to define how the security of the network is set up. Other infrastructure elements such as load balancers, databases, etc. may also be provisioned. The infrastructure may evolve incrementally as more infrastructure elements are desired and / or added.

[0053] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques may enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across a wide variety of geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure onto which the code will be deployed must first be set up. In some examples, provisioning may be done manually, and a provisioning tool may be utilized to provision the resources, and / or once the infrastructure is provisioned, a deployment tool may be utilized to deploy the code.

[0054] FIG. 6 is a block diagram 600 illustrating an example pattern of an IaaS architecture, according to at least one embodiment. A service operator 602 may be communicatively coupled to a secure host tenancy 604, which may include a virtual cloud network (VCN) 606 and a secure host subnet 608. In some examples, the service operator 602 may employ one or more client computing devices. The client computing devices may be portable handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and supporting Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing device may be a workstation computer running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing devices may be other electronic devices capable of communicating over VCN 606 and / or an Internet-accessible network, such as thin-client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox game consoles with or without Kinect® gesture input devices), and / or personal messaging devices.

[0055] The VCN 606 may include a local peering gateway (LPG) 610. The LPG 610 may be communicatively coupled to a secure shell (SSH) VCN 612 via an LPG 610 included in the SSH VCN 612. The SSH VCN 612 may include an SSH subnet 614, which may be communicatively coupled to a control plane VCN 616 via an LPG 610 included in the control plane VCN 616. The SSH VCN 612 may also be communicatively coupled to a data plane VCN 618 via the LPG 610. The control plane VCN 616 and the data plane VCN 618 may be included in a service tenancy 619, which may be owned and / or operated by the IaaS provider.

[0056] The control plane VCN 616 may include a control plane de-militarized zone (DMZ) tier 620 that serves as a perimeter network (e.g., a portion of an enterprise network between an enterprise intranet and an external network). DMZ-based servers may have limited responsibilities and help mitigate security breaches. In addition, the DMZ tier 620 may include one or more load balancer (LB) subnets 622, a control plane app tier 624 that may include an app subnet 626, and a control plane data tier 628 that may include a database (DB) subnet 630 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 622 included in the control plane DMZ tier 620 may be communicatively coupled to the app subnet 626 included in the control plane app tier 624 and an Internet gateway 634 that may be included in the control plane VCN 616, and the app subnet 626 may be communicatively coupled to the DB subnet 630, a service gateway 636, and a network address translation (NAT) gateway 638 included in the control plane data tier 628. The control plane VCN 616 may include a service gateway 636 and a NAT gateway 638.

[0057] The control plane VCN 616 may include a data plane mirror app layer 640, which may include an app subnet 626. The app subnet 626 included in the data plane mirror app layer 640 may include a virtual network interface controller (VNIC) 642, which may run a compute instance 644. The compute instance 644 may communicatively couple the app subnet 626 of the data plane mirror app layer 640 to the app subnet 626, which may be included in the data plane app layer 646.

[0058] The data plane VCN 618 may include a data plane app layer 646, a data plane DMZ layer 648, and a data plane data layer 650. The data plane DMZ layer 648 may include a LB subnet 622 that may be communicatively coupled to an app subnet 626 of the data plane app layer 646 and an Internet gateway 634 of the data plane VCN 618. The app subnet 626 may be communicatively coupled to a service gateway 636 of the data plane VCN 618 and a NAT gateway 638 of the data plane VCN 618. The data plane data layer 650 may also include a DB subnet 630 that may be communicatively coupled to the app subnet 626 of the data plane app layer 646.

[0059] The Internet gateways 634 of the control plane VCNs 616 and data plane VCNs 618 may be communicatively coupled to a metadata management service 652, which may be communicatively coupled to the public Internet 654. The public Internet 654 may be communicatively coupled to a NAT gateway 638 of the control plane VCNs 616 and data plane VCNs 618. The service gateways 636 of the control plane VCNs 616 and data plane VCNs 618 may be communicatively coupled to cloud services 656.

[0060] In some examples, a service gateway 636 of a control plane VCN 616 or a data plane VCN 618 may make an application programming interface (API) call to a cloud service 656 without traversing the public Internet 654. An API call from a service gateway 636 to a cloud service 656 may be one-way. That is, the service gateway 636 may make an API call to a cloud service 656, and the cloud service 656 may send the requested data to the service gateway 636. However, the cloud service 656 may not initiate the API call to the service gateway 636.

[0061] In some examples, secure host tenancy 604 may be directly connected to service tenancy 619, which may otherwise be separate. Secure host subnet 608 may communicate with SSH subnet 614 through LPG 610, which may allow bidirectional communication through otherwise separate systems. Connecting secure host subnet 608 to SSH subnet 614 may give secure host subnet 608 access to other entities in service tenancy 619.

[0062] The control plane VCN 616 may enable users of a service tenancy 619 to set up or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 616 may be deployed or otherwise used in the data plane VCN 618. In some examples, the control plane VCN 616 may be separate from the data plane VCN 618, and the data plane mirror app layer 640 of the control plane VCN 616 may communicate with the data plane app layer 646 of the data plane VCN 618 via a VNIC 642, which may be included in the data plane mirror app layer 640 and the data plane app layer 646.

[0063] In some examples, a user or customer of the system may make a request, such as, for example, a create, read, update, or delete (CRUD) operation, through the public Internet 654, which may communicate the request to a metadata management service 652. The metadata management service 652 may communicate the request to the control plane VCN 616 through an Internet gateway 634. The request may be received by a LB subnet 622 included in the control plane DMZ layer 620. The LB subnet 622 may determine that the request is valid, and in response to this determination, the LB subnet 622 may send the request to an app subnet 626 included in the control plane app layer 624. If the request is validated and uses a call to the public Internet 654, the call to the public Internet 654 may be sent to a NAT gateway 638, which may make the call to the public Internet 654. Memory that may be desired to be stored by the request may be stored in the DB subnet 630.

[0064] In some examples, the data plane mirror app layer 640 may facilitate direct communication between the control plane VCN 616 and the data plane VCN 618. For example, it may be desired that a configuration change, update, or other appropriate modification be applied to resources included in the data plane VCN 618. Through the VNIC 642, the control plane VCN 616 may communicate directly with resources included in the data plane VCN 618, thereby performing the change, update, or other appropriate modification to the configuration.

[0065] In some embodiments, the control plane VCN 616 and the data plane VCN 618 may be included in the service tenancy 619. In this case, a user or customer of the system may not own or operate either the control plane VCN 616 or the data plane VCN 618. Instead, an IaaS provider may own or operate the control plane VCN 616 and the data plane VCN 618, both of which may be included in the service tenancy 619. This embodiment may allow for network isolation that may prevent a user or customer from interacting with other users' or other customers' resources. This embodiment may also allow a user or customer of the system to store databases informally without having to rely on the public Internet 654 for storage, which may not have the desired level of security.

[0066] In another embodiment, the LB subnet 622 included in the control plane VCN 616 may be configured to receive signals from the service gateway 636. In this embodiment, the control plane VCN 616 and the data plane VCN 618 may be configured to be called by the IaaS provider's customers without calling the public Internet 654. The IaaS provider's customers may desire this embodiment because databases used by the customers may be controlled by the IaaS provider and may be stored on the service tenancy 619, which may be isolated from the public Internet 654.

[0067] 7 is a block diagram 700 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 702 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 704 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network (VCN) 706 (e.g., VCN 606 of FIG. 6) and a secure host subnet 708 (e.g., secure host subnet 608 of FIG. 6). The VCN 706 may include a local peering gateway (LPG) 710 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to a secure shell (SSH) VCN 712 (e.g., SSH VCN 612 of FIG. 6) via an LPG 610 included in the SSH VCN 712. SSH VCN 712 may include an SSH subnet 714 (e.g., SSH subnet 614 in FIG. 6), which may be communicatively coupled to a control plane VCN 716 (e.g., control plane VCN 616 in FIG. 6) via an LPG 710 included in the control plane VCN 716. The control plane VCN 716 may be included in a service tenancy 719 (e.g., service tenancy 619 in FIG. 6), and the data plane VCN 718 (e.g., data plane VCN 618 in FIG. 6) may be included in a customer tenancy 721, which may be owned or operated by a user or customer of the system.

[0068] The control plane VCN 716 may include a control plane DMZ tier 720 (e.g., control plane DMZ tier 620 of FIG. 6 ) that may include a LB subnet 722 (e.g., LB subnet 622 of FIG. 6 ), a control plane app tier 724 (e.g., control plane app tier 624 of FIG. 6 ) that may include an app subnet 726 (e.g., app subnet 626 of FIG. 6 ), and a control plane data tier 728 (e.g., control plane data tier 628 of FIG. 6 ) that may include a database (DB) subnet 730 (e.g., similar to DB subnet 630 of FIG. 6 ). The LB subnet 722 included in the control plane DMZ layer 720 may be communicatively coupled to an app subnet 726 included in the control plane app layer 724 and to an Internet gateway 734 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 716, and the app subnet 726 may be communicatively coupled to a DB subnet 730 included in the control plane data layer 728, to a service gateway 736 (e.g., service gateway in FIG. 6 ), and to a network address translation (NAT) gateway 738 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 716 may include the service gateway 736 and the NAT gateway 738.

[0069] The control plane VCN 716 may include a data plane mirror app layer 740 (e.g., data plane mirror app layer 640 of FIG. 6 ), which may include an app subnet 726. The app subnet 726 included in the data plane mirror app layer 740 may include a virtual network interface controller (VNIC) 742 (e.g., VNIC 642) that may run a compute instance 744 (e.g., similar to compute instance 644 of FIG. 6 ). The compute instance 744 may facilitate communication between the app subnet 726 of the data plane mirror app layer 740 and the app subnet 726 that may be included in the data plane app layer 746 (e.g., data plane app layer 646 of FIG. 6 ) via the VNIC 742 included in the data plane mirror app layer 740 and the VNIC 742 included in the data plane app layer 746.

[0070] An Internet gateway 734 included in the control plane VCN 716 may be communicatively coupled to a metadata management service 752 (e.g., metadata management service 652 of FIG. 6), which may be communicatively coupled to a public Internet 754 (e.g., public Internet 654 of FIG. 6). The public Internet 754 may be communicatively coupled to a NAT gateway 738 included in the control plane VCN 716. A service gateway 736 included in the control plane VCN 716 may be communicatively coupled to cloud services 756 (e.g., cloud services 656 of FIG. 6).

[0071] In some examples, the data plane VCN 718 may be included in the customer tenancy 721. In this case, the IaaS provider may provide a control plane VCN 716 for each customer, and the IaaS provider may set up a unique compute instance 744 included in the service tenancy 719 for each customer. Each compute instance 744 may enable communication between the control plane VCN 716 included in the service tenancy 719 and the data plane VCN 718 included in the customer tenancy 721. The compute instance 744 may enable resources provisioned in the control plane VCN 716 included in the service tenancy 719 to be deployed or otherwise used in the data plane VCN 718 included in the customer tenancy 721.

[0072] In another example, an IaaS provider customer may have a database that resides in customer tenancy 721. In this example, control plane VCN 716 may include a data plane mirror app layer 740 that may include app subnet 726. The data plane mirror app layer 740 may reside in data plane VCN 718, but the data plane mirror app layer 740 may not reside in data plane VCN 718. That is, the data plane mirror app layer 740 may have access to customer tenancy 721, but the data plane mirror app layer 740 may not reside in data plane VCN 718 or be owned or operated by the IaaS provider customer. The data plane mirror app layer 740 may be configured to make calls to the data plane VCN 718, but may not be configured to make calls to any entities included in the control plane VCN 716. A customer may desire to deploy or otherwise use resources in a data plane VCN 718 that are provisioned in a control plane VCN 716, and the data plane mirror app layer 740 may facilitate the desired deployment or other use of the customer's resources.

[0073] In some embodiments, the IaaS provider's customer may apply filters to the data plane VCN 718. In this embodiment, the customer may determine what the data plane VCN 718 can access, and the customer may limit access from the data plane VCN 718 to the public Internet 754. The IaaS provider may not be able to filter or otherwise control access of the data plane VCN 718 to any external networks or databases. Filters and controls applied by the customer on the data plane VCN 718 included in the customer tenancy 721 can help isolate the data plane VCN 718 from other customers and the public Internet 754.

[0074] In some embodiments, cloud services 756 may be invoked by service gateway 736 to access services that may not be on the public internet 754, on the control plane VCN 716, or on the data plane VCN 718. The connection between cloud services 756 and the control plane VCN 716 or data plane VCN 718 may not be live or continuous. Cloud services 756 may be on different networks owned or operated by an IaaS provider. Cloud services 756 may be configured to receive calls from service gateway 736 and may not be configured to receive calls from the public internet 754. Some cloud services 756 may be isolated from other cloud services 756, and control plane VCN 716 may be isolated from cloud services 756 that may not be in the same region as control plane VCN 716. For example, control plane VCN 716 may be located in “Region 1” and cloud service “Deployment 6” may be located in Region 1 and “Region 2.” If a call to Deployment 6 is made by a service gateway 736 included in a control plane VCN 716 located in Region 1, the call may be sent to Deployment 6 in Region 1. In this example, control plane VCN 716, or Deployment 6 in Region 1, may not be communicatively coupled to or otherwise in communication with Deployment 6 in Region 2.

[0075] 8 is a block diagram 800 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 802 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 804 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network (VCN) 806 (e.g., VCN 606 of FIG. 6) and a secure host subnet 808 (e.g., secure host subnet 608 of FIG. 6). VCN 806 may include an LPG 810 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to an SSH VCN 812 (e.g., SSH VCN 612 of FIG. 6) via an LPG 810 included in SSH VCN 812. SSH VCN 812 may include an SSH subnet 814 (e.g., SSH subnet 614 in FIG. 6), which may be communicatively coupled to a control plane VCN 816 (e.g., control plane VCN 616 in FIG. 6) via an LPG 810 included in the control plane VCN 816, and to a data plane VCN 818 (e.g., data plane 618 in FIG. 6) via an LPG 810 included in the data plane VCN 818. The control plane VCN 816 and the data plane VCN 818 may be included in a service tenancy 819 (e.g., service tenancy 619 in FIG. 6).

[0076] The control plane VCN 816 may include a control plane DMZ tier 820 (e.g., control plane DMZ tier 620 of FIG. 6 ) that may include a load balancer (LB) subnet 822 (e.g., LB subnet 622 of FIG. 6 ), a control plane app tier 824 (e.g., control plane app tier 624 of FIG. 6 ) that may include an app subnet 826 (e.g., similar to app subnet 626 of FIG. 6 ), and a control plane data tier 828 (e.g., control plane data tier 628 of FIG. 6 ) that may include a DB subnet 830. The LB subnet 822 included in the control plane DMZ layer 820 may be communicatively coupled to an app subnet 826 included in the control plane app layer 824 and to an Internet gateway 834 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 816, and the app subnet 826 may be communicatively coupled to a DB subnet 830 included in the control plane data layer 828 and to a service gateway 836 (e.g., service gateway in FIG. 6 ) and a network address translation (NAT) gateway 838 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0077] The data plane VCN 818 may include a data plane app layer 846 (e.g., data plane app layer 646 of FIG. 6 ), a data plane DMZ layer 848 (e.g., data plane DMZ layer 648 of FIG. 6 ), and a data plane data layer 850 (e.g., data plane data layer 650 of FIG. 6 ). The data plane DMZ layer 848 may include a LB subnetwork 822 that may be communicatively coupled to a trusted app subnetwork 860 and an untrusted app subnetwork 862 of the data plane app layer 846 and an Internet gateway 834 included in the data plane VCN 818. The trusted app subnetwork 860 may be communicatively coupled to a service gateway 836 included in the data plane VCN 818, a NAT gateway 838 included in the data plane VCN 818, and a DB subnetwork 830 included in the data plane data layer 850. The untrusted app subnetwork 862 may be communicatively coupled to a service gateway 836 included in the data plane VCN 818 and a DB subnetwork 830 included in the data plane data layer 850. The data plane data layer 850 may include a DB subnet 830 that may be communicatively coupled to a service gateway 836 included in the data plane VCN 818.

[0078] The untrusted app subnet 862 may include one or more primary VNICs 864(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 866(1)-(N). Each tenant VM 866(1)-(N) may be communicatively coupled to a respective app subnet 867(1)-(N), which may be included in a respective container egress VCN 868(1)-(N), which may be included in a respective customer tenancy 870(1)-(N). Each secondary VNIC 872(1)-(N) may facilitate communication between the untrusted app subnet 862 included in the data plane VCN 818 and the app subnet included in the container egress VCN 868(1)-(N). Each container egress VCN 868(1)-(N) may include a NAT gateway 838, which may be communicatively coupled to the public Internet 854 (e.g., public Internet 654 in FIG. 6).

[0079] An Internet gateway 834 included in the control plane VCN 816 and in the data plane VCN 818 may be communicatively coupled to a metadata management service 852 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public Internet 854. The public Internet 854 may be communicatively coupled to a NAT gateway 838 included in the control plane VCN 816 and in the data plane VCN 818. A service gateway 836 included in the control plane VCN 816 and in the data plane VCN 818 may be communicatively coupled to cloud services 856.

[0080] In some embodiments, data plane VCN 818 may be integrated with customer tenancies 870. This integration may be useful or desirable for an IaaS provider's customers in some cases, such as when run-time support of code is desired. A customer may provide code to be executed that may be disruptive, communicate with other customer resources, or otherwise cause undesirable effects. In response, the IaaS provider may determine whether to execute the code that the customer provided to the IaaS provider.

[0081] In some examples, an IaaS provider's customer may request to grant temporary network access to the IaaS provider to attach a function to data plane layer app 846. Code to perform the function may be executed in VM 866(1)-(N), and the code may not be configured to run anywhere else on data plane VCN 818. Each VM 866(1)-(N) may be connected to one customer tenancy 870. Each container 871(1)-(N) included in VM 866(1)-(N) may be configured to execute code. In this case, there may be double isolation (e.g., container 871(1)-(N) may execute code, and container 871(1)-(N) may be included in at least VM 866(1)-(N) included in untrusted app subnet 862), which may help prevent erroneous or otherwise undesirable code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 871(1)-(N) may be communicatively coupled to customer tenancy 870 and may be configured to send or receive data from customer tenancy 870. Containers 871(1)-(N) may not be configured to send or receive data from any other entities in data plane VCN 818. Once code execution is complete, the IaaS provider may kill or otherwise discard containers 871(I)-(N).

[0082] In some embodiments, trusted app subnet 860 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 860 may be communicatively coupled to DB subnet 830 and may be configured to perform CRUD operations on DB subnet 830. Untrusted app subnet 862 may be communicatively coupled to DB subnet 830, but in this embodiment, untrusted app subnet 862 may be configured to perform read operations on DB subnet 830. Containers 871(1)-(N) that may be included in each customer's VMs 866(1)-(N) and that may execute code from the customer may not be communicatively coupled to DB subnet 830.

[0083] In other embodiments, the control plane VCN 816 and the data plane VCN 818 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 816 and the data plane VCN 818. However, communication may occur indirectly through at least one method. An LPG 810 may be established by an IaaS provider that may facilitate communication between the control plane VCN 816 and the data plane VCN 818. In another example, the control plane VCN 816 or the data plane VCN 818 may make a call to a cloud service 856 through a service gateway 836. For example, a call from the control plane VCN 816 to the cloud service 856 may include a request for a service that may communicate with the data plane VCN 818.

[0084] 9 is a block diagram 900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 902 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 904 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network (VCN) 906 (e.g., VCN 606 of FIG. 6) and a secure host subnet 908 (e.g., secure host subnet 608 of FIG. 6). VCN 906 may include an LPG 910 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to an SSH VCN 912 (e.g., SSH VCN 612 of FIG. 6) via an LPG 910 included in SSH VCN 912. SSH VCN 912 may include an SSH subnet 914 (e.g., SSH subnet 614 in FIG. 6), which may be communicatively coupled to a control plane VCN 916 (e.g., control plane VCN 616 in FIG. 6) via an LPG 910 included in the control plane VCN 916, and to a data plane VCN 918 (e.g., data plane 618 in FIG. 6) via an LPG 910 included in the data plane VCN 918. The control plane VCN 916 and the data plane VCN 918 may be included in a service tenancy 919 (e.g., service tenancy 619 in FIG. 6).

[0085] The control plane VCN 916 may include a control plane DMZ layer 920 (e.g., control plane DMZ layer 620 of FIG. 6 ) that may include a LB subnet 922 (e.g., LB subnet 622 of FIG. 6 ), a control plane app layer 924 (e.g., control plane app layer 624 of FIG. 6 ) that may include an app subnet 926 (e.g., app subnet 626 of FIG. 6 ), and a control plane data layer 928 (e.g., control plane data layer 628 of FIG. 6 ) that may include a DB subnet 930 (e.g., DB subnet 830 of FIG. 8 ). The LB subnet 922 included in the control plane DMZ layer 920 may be communicatively coupled to an app subnet 926 included in the control plane app layer 924 and to an Internet gateway 934 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 916, and the app subnet 926 may be communicatively coupled to a DB subnet 930 included in the control plane data layer 928 and to a service gateway 936 (e.g., service gateway in FIG. 6 ) and a network address translation (NAT) gateway 938 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0086] The data plane VCN 918 may include a data plane app layer 946 (e.g., data plane app layer 646 in FIG. 6 ), a data plane DMZ layer 948 (e.g., data plane DMZ layer 648 in FIG. 6 ), and a data plane data layer 950 (e.g., data plane data layer 650 in FIG. 6 ). The data plane DMZ layer 948 may include a LB subnet 922 that may be communicatively coupled to a trusted app subnet 960 (e.g., trusted app subnet 860 in FIG. 8 ) and an untrusted app subnet 962 (e.g., untrusted app subnet 862 in FIG. 8 ) of the data plane app layer 946 and an Internet gateway 934 included in the data plane VCN 918. The trusted app subnet 960 may be communicatively coupled to a service gateway 936 included in the data plane VCN 918, a NAT gateway 938 included in the data plane VCN 918, and a DB subnet 930 included in the data plane data layer 950. The untrusted app subnet 962 may be communicatively coupled to a service gateway 936 included in the data plane VCN 918 and a DB subnet 930 included in the data plane data layer 950. The data plane data layer 950 may include the DB subnet 930, which may be communicatively coupled to a service gateway 936 included in the data plane VCN 918.

[0087] The untrusted app subnet 962 may include primary VNICs 964(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N) residing within the untrusted app subnet 962. Each tenant VM 966(1)-(N) may execute code in a respective container 967(1)-(N), which may be communicatively coupled to an app subnet 926, which may be included in a data plane app layer 946, which may be included in a container egress VCN 968. Each secondary VNIC 972(1)-(N) may facilitate communication between the untrusted app subnet 962, which may be included in the data plane VCN 918, and the app subnet, which may be included in the container egress VCN 968. The container egress VCN may include a NAT gateway 938, which may be communicatively coupled to the public Internet 954 (e.g., public Internet 654 of FIG. 6).

[0088] An internet gateway 934 included in the control plane VCN 916 and in the data plane VCN 918 may be communicatively coupled to a metadata management service 952 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public internet 954. The public internet 954 may be communicatively coupled to a NAT gateway 938 included in the control plane VCN 916 and in the data plane VCN 918. A service gateway 936 included in the control plane VCN 916 and in the data plane VCN 918 may be communicatively coupled to cloud services 956.

[0089] In some examples, the pattern illustrated by the architecture of block diagram 900 of FIG. 9 may be considered an exception to the pattern illustrated by the architecture of block diagram 800 of FIG. 8 and may be desirable for a customer of an IaaS provider when the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected area). Each container 967(1)-(N) contained in VM 966(1)-(N) for each customer may be accessed in real time by the customer. The containers 967(1)-(N) may be configured to make calls to each secondary VNIC 972(1)-(N) contained in app subnet 926 of data plane app tier 946 that may be contained in container egress VCN 968. The secondary VNIC 972(1)-(N) may send the call to NAT gateway 938, which may send the call to the public Internet 954. In this example, containers 967(1)-(N) that may be accessed in real time by a customer may be isolated from control plane VCN 916 and may be isolated from other entities included in data plane VCN 918. Containers 967(1)-(N) may also be isolated from resources from other customers.

[0090] In another example, a customer may invoke cloud service 956 using container 967(1)-(N). In this example, the customer may execute code in container 967(1)-(N) that requests a service from cloud service 956. Container 967(1)-(N) may send the request to secondary VNIC 972(1)-(N), which may send the request to a NAT gateway, which may send the request to public internet 954. Public internet 954 may send the request to LB subnet 922, which is included in control plane VCN 916, via internet gateway 934. In response to determining that the request is valid, LB subnet 926 may send the request to app subnet 926, which may send the request to cloud service 956 via service gateway 936.

[0091] It should be noted that the IaaS architectures 600, 700, 800, 900 depicted in the figures may have components other than those depicted. Additionally, the depicted embodiments are only some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS systems may have more or fewer components than depicted, may combine two or more components, or may have different configurations or arrangements of components.

[0092] In certain embodiments, the IaaS system described herein may include a suite of application, middleware, and database services products that are self-service, subscription-based, elastically scalable, reliable, highly available, and securely delivered to customers. One example of such an IaaS system is Oracle Cloud Infrastructure (OCI), offered by the assignee of the present application.

[0093] 10 illustrates an exemplary computer system 1000 on which various embodiments of the present disclosure may be implemented. The system 1000 may be used to implement any of the computer systems described above. As illustrated, the computer system 1000 includes a processing unit 1004 that communicates with several peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and a system memory 1010.

[0094] Bus subsystem 1002 provides a mechanism for allowing the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is shown diagrammatically as a bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0095] The processing unit 1004, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core processors or multi-core processors. In particular embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, with a single-core processor or a multi-core processor included in each processing unit. In other embodiments, the processing unit 1004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into one chip.

[0096] In various embodiments, the processing unit 1004 may execute various programs in response to program code and may maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the processor 1004 and / or the storage subsystem 1018. Through appropriate programming, the processor 1004 may provide the various functions described above. The computer system 1000 may additionally include a processing acceleration unit 1006, which may include a digital signal processor (DSP), special purpose processor, and / or others.

[0097] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, a motion sensing and / or gesture recognition device such as a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and spoken commands. User interface input devices may also include an eye gesture recognition device such as a Google Glass® blink detector that detects eye movements from a user (e.g., "blinking" during picture taking and / or menu selection) and translates the eye gesture as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (eg, the Siri® navigator) through voice commands.

[0098] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, medical ultrasound, etc. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, etc.

[0099] User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, when the term "output device" is used, it is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. For example, user interface output devices may include various display devices that visually convey textual, graphical, and audio / visual information, such as, but not limited to, monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0100] Computer system 1000 may include a storage subsystem 1018 that includes software elements currently shown as being within system memory 1010. System memory 1010 may store program instructions loadable and executable on processing unit 1004, as well as data generated during the execution of these programs.

[0101] Depending on the configuration and type of computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 1004. In some implementations, the system memory 1010 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1000, such as during start-up, may typically be stored in ROM. By way of example and not limitation, the system memory 1010 also illustrates application programs 1012, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like, program data 1014, and an operating system 1016. By way of example, operating system 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux® operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS operating systems.

[0102] The storage subsystem 1018 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the above-described functionality may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with the present disclosure.

[0103] Storage subsystem 1000 may also include a computer readable storage medium reader 1020 that may be further connected to a computer readable storage medium 1022. Along with, and optionally in combination with, the system memory 1010, computer readable storage medium 1022 may comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for containing, storing, transmitting, and retrieving computer readable information on a temporary and / or more permanent basis.

[0104] The computer readable storage medium 1022 containing the code or portions of code may also include any suitable medium known or used in the art, including, but not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media, implemented in any manner or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable media. This may also include non-tangible computer readable media, such as a data signal, data transmission, or any other medium that can be used to transmit the desired information and that can be accessed by the computing system 1000.

[0105] By way of example, the computer readable storage medium 1022 may include hard disk drives that read from and write to non-removable, non-volatile magnetic media, magnetic disk drives that read from and write to removable, non-volatile magnetic disks, and optical disk drives that read from and write to removable, non-volatile optical disks, such as CD ROMs, DVDs, and Blu-Ray disks, or other optical media. The computer readable storage medium 1022 may include, but is not limited to, Zip drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer readable storage medium 1022 may also include flash memory-based SSDs, enterprise flash drives, solid state drives (SSDs) based on non-volatile memory such as solid state ROM, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer system 1000.

[0106] The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for transmitting and receiving data between other systems and the computer system 1000. For example, the communications subsystem 1024 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1024 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1024 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0107] In some embodiments, the communications subsystem 1024 may also receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. on behalf of one or more users who may be using the computer system 1000.

[0108] By way of example, the communications subsystem 1024 may be configured to receive data feeds 1026 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0109] Additionally, the communications subsystem 1024 may also be configured to receive data in the form of continuous data streams, which may include event streams 1028 of real-time events and / or event updates 1030, which may be continuous or infinite in nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0110] The communications subsystem 1024 may also be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1000.

[0111] The computer system 1000 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0112] Due to the ever-changing nature of computers and networks, the illustration of computer system 1000 shown in the figures is intended as only one specific example. Many other configurations are possible having more or fewer components than the system shown in the figures. For example, customized hardware may be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other ways and / or methods for implementing various embodiments.

[0113] The term cloud services is commonly used to refer to services that are made available on demand (e.g., via a subscription model) to users or customers by a cloud service provider (CSP) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. Thus, customers can utilize cloud services provided by a CSP without having to purchase separate hardware and software resources for those services. Cloud services are designed to provide customers with easy and scalable access to applications and computing resources without the need for subscribing customers to invest in procuring the infrastructure used to provide those services.

[0114] There are several cloud service providers offering different types of cloud services. There are a variety of different types or models of cloud services, including Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Infrastructure-as-a-Service (IaaS), etc.

[0115] A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer subscribes or registers for a service offered by a CSP, a tenancy or account is created for that customer. Through this account, the customer can then access one or more subscribed cloud resources associated with the account.

[0116] As mentioned above, Infrastructure as a Service (IaaS) is a particular type of cloud computing service. In the IaaS model, the CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0117] CSPI may include interconnected high performance computing resources including various host machines, memory resources, and network resources that form a physical network, also referred to as a backbone or underlay network. CSPI resources may be distributed across one or more data centers that may be geographically distributed across one or more geographic regions. Virtualization software may be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network provides the basis for creating one or more overlay or virtual networks on top of the physical network. The virtual or overlay network may include one or more virtual cloud networks (VCNs). The virtual network is realized using software virtualization techniques (e.g., hypervisors, functions performed by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that realize one or more functions performed by the NVDs, and other mechanisms) to create a network abstraction layer that can operate on top of the physical network. The virtual network may take various forms including peer-to-peer networks, IP networks, etc. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often called virtual Layer 3 networking or overlay Layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LANs (VXLAN-IETF RFC7348), Virtual Private Networks (VPNs) (e.g. MPLS Layer 3 Virtual Private Networks (RFC4364)), VMware's NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0118] In the case of IaaS, the infrastructure (CSPI) provided by the CSP may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, the cloud computing service provider may host the infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.) that accompany those infrastructure components. Thus, these services may be policy-driven, so that IaaS users may be able to implement policies that drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and power as well as storage capacity in a flexible virtual network that can be securely accessed from various network locations, such as from the customer's on-premise network. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a secure, isolated partition within the CSP in which the customer can create, organize, and manage their cloud resources.

[0119] Customers can build their own virtual networks using compute, memory, and networking resources provided by CSPI. On these virtual networks, they can deploy one or more customer resources or workloads, such as compute instances. For example, customers can build one or more customizable private virtual networks, called Virtual Cloud Networks (VCNs), using resources provided by CSPI. On the customer VCNs, customers can deploy one or more customer resources, such as compute instances. The compute instances can take the form of virtual machines, bare metal instances, etc. Thus, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, virtual hosted environment. Customers do not manage or control the underlying physical resources provided by CSPI, but rather control the operating systems, storage, and deployed applications, and in some cases have limited control over some networking components (e.g., firewalls).

[0120] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0121] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions and safeguards are taken in CSPI to ensure that each tenant's data remains isolated and invisible to other tenants.

[0122] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to a physical network and communicates bidirectionally with the network to which it is connected. Network endpoints in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), and the like. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), and the like. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as overlay layer 2 addresses (e.g., an overlay MAC address) and overlay layer 3 addresses (e.g., an overlay IP address). Network overlays enable flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software that implements the virtual network's control plane). Thus, unlike physical networks, in a virtual network, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on top of physical networks, communication between components in a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the backbone network, and vice versa. These mappings are then used to facilitate communications. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0123] Thus, a physical address (e.g., a physical IP address) is associated with a component in a physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity in a virtual network. Physical and overlay IP addresses are both types of real IP addresses. They are distinct from virtual IP addresses, which are mapped to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses.

[0124] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include components in a physical or backbone network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In certain embodiments, the CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a localized geographic area that includes one or more data centers. Regions are generally independent of each other and may be separated by vast distances (e.g., across countries or even continents). For example, a first region may be in Australia, another in Japan, yet another in India, etc. CSPI resources are divided between regions such that each region has its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers and related infrastructure, etc.), storage resources (e.g., block volume storage, file storage, object storage, archival storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connections to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region typically has multiple paths connecting it to other regions in the realm.

[0125] Generally, an application is deployed in the region where it is most heavily used (i.e., on the infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed in different regions for a variety of reasons, such as redundancy to reduce the risk of a region-wide event such as a large weather system or earthquake, redundancy to meet various requirements such as legal jurisdictions, tax territories, and other business or social criteria.

[0126] Data centers within a region may be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located within a region. A region may be composed of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0127] ADs in a region are isolated from each other, fault tolerant, and configured to be unlikely to fail simultaneously. This is achieved by ADs not sharing critical infrastructure resources such as networking, physical cables, cable paths, and cable entry points, so that a failure in one AD in a region is unlikely to affect the availability of other ADs in the same region. ADs in the same region can be connected to each other by low-latency, high-bandwidth networks to provide highly available connections to other networks (e.g., the Internet, customer on-premise networks, etc.), and multiple ADs can be used to build replicated systems for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by an IaaS provider evolves, more regions and ADs can be added along with additional capacity. Traffic between availability domains is typically encrypted.

[0128] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share data. Regions in the same realm may communicate with each other, but regions in different realms cannot communicate with each other. A CSP's customer's tenancy or account exists in one realm and can span one or more regions that belong to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account is created for the customer in a customer-specified region (called the "home" region) within the realm. The customer can extend their tenancy to one or more other regions within the realm. The customer cannot access regions that are not within the realm in which their tenancy resides.

[0129] An IaaS provider may offer multiple realms, each targeted to a particular set of customers or users. For example, a commercial realm may be offered to commercial customers. As another example, a realm may be offered to a particular country or customers in that country. As yet another example, a government realm may be offered to a government, and so on. For example, a government realm may be targeted to a particular government and may have a higher security level than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for a commercial region, as well as two realms for a government cloud region (e.g., FedRAMP certified and IL5 certified).

[0130] In certain embodiments, an AD may be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains can distribute compute instances so that they are not on the same physical hardware within an AD. This is known as anti-affinity. A fault domain refers to a set of hardware components (computers, switches, and others) that share a single point of failure. Compute pools are logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in certain embodiments, each AD includes three fault domains. Fault domains act as logical data centers within an AD.

[0131] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for that customer and associated with that customer's tenancy. Customers can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted in the cloud by CSPI is called a Virtual Cloud Network (VCN). A customer can set up one or more Virtual Cloud Networks (VCNs) using CSPI resources allocated to them. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN may include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints ("public endpoints") over a public network such as the Internet, can communicate with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), can communicate with the customer's on-premise data center or network, can communicate with service endpoints and other types of endpoints.

[0132] CSPs may use CSPIs to provide various services. In some examples, customers of CSPIs themselves may act as service providers and provide services using CSPI resources. Service providers may expose service endpoints characterized by identification information (e.g., IP addresses, DNS names and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing a service endpoint exposed by a particular service for that particular service. These service endpoints are generally endpoints that are publicly accessible by users over a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0133] In certain embodiments, a service provider may expose a service through a service endpoint (sometimes referred to as a service endpoint). Customers of the service may then access the service using the service endpoint. In certain implementations, a service endpoint provided for a service may be accessible by multiple customers intending to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer such that only the customer may access the service using the dedicated service endpoint.

[0134] In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses (e.g., 10.0 / 16) that are assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN resides within a region but can span one or more or all of the region's availability domains. A gateway is a virtual interface configured for a VCN that enables traffic to communicate between the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication between different types of endpoints.

[0135] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a building block or subdivision that can be created within a VCN. A VCN may have one or more subnets. Each subnet in a VCN is associated with a contiguous range of overlay IP addresses that do not overlap with other subnets in that VCN and that represent a subset of address space within the VCN's address space (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24).

[0136] Each compute instance is associated with a virtual network interface card (VNIC) that allows the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a layer 2 port on a switch. A VNIC is attached to a compute instance and a subnet in a VCN. A VNIC associated with a compute instance allows the compute instance to be part of a subnet of a VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, communicate with endpoints in a different subnet in the VCN, or communicate with endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. For a subnet that includes a set of compute instances, the subnet includes VNICs that correspond to the set of compute instances, and each VNIC is attached to a compute instance in the set of compute instances.

[0137] Each compute instance is assigned a private overlay IP address through the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses that do not overlap with other subnets in the VCN and that represent a subset of address space within the VCN's address space (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24). For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0138] In certain embodiments, a compute instance may be assigned additional overlay IP addresses, such as one or more public IP addresses if in a public subnet, in addition to the private overlay IP address. These multiple addresses may be assigned on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created during instance launch and associated with an overlay private IP address assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in a subnet in the same VCN as the primary VNIC, or in a different subnet in the same or a different VCN.

[0139] Compute instances may optionally be assigned a public IP address if they are in a public subnet. Subnets can be specified as either public or private when creating the subnet. A private subnet means that resources in the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources in the subnet and associated VNICs can have public IP addresses. Customers can specify whether a subnet resides in one availability domain or across multiple availability domains within a region or realm.

[0140] As mentioned above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply VR) configured for a VCN enables communication between subnets of the VCN. For a subnet in a VCN, the VR represents a logical gateway for that subnet that enables the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets inside the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, which potentially has an unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. Thus, a VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VRs are also connected to the various gateways configured for the VCN. In certain embodiments, a particular overlay IP address from a subnet's overlay IP address range is reserved for the ports of the VCN VR of that subnet. For example, consider a VCN with two subnets, with address ranges 10.0 / 16 and 10.1 / 16 associated with each. For a first subnet in the VCN with address range 10.0 / 16, an address from this range is reserved for the ports of the VCN VR of that subnet. In some examples, a first IP address from the range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for the ports of the VCN VR of that subnet. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port of that second subnet with IP address 10.1.0.1.A VCN VR has a different IP address for each subnet in the VCN.

[0141] In some other embodiments, each subnet in a VCN may have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, a first IP address from a range of IP addresses associated with the subnet. A VNIC in a subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, the VR is the ingress / egress point for that subnet. A VR associated with a subnet in a VCN can communicate with other VRs associated with other subnets in the VCN. A VR can also communicate with a gateway associated with the VCN. The VR functions of a subnet are running on or performed by one or more NVDs that perform the VNIC functions of the VNICs in the subnet.

[0142] Route tables, security rules, and DHCP options may be configured for a VCN. A route table is a virtual route table for a VCN and contains rules for routing traffic from subnets inside the VCN to destinations outside the VCN via gateways or specially configured instances. You can customize a VCN's route table to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically given to an instance when it boots up.

[0143] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the type of traffic (e.g., based on protocol and port) that is allowed to flow in and out of instances in the VCN. A customer can choose whether a given rule is stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by setting up a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be realized using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources in that group. On the other hand, a security list contains rules that apply to all resources in any subnet that uses the security list. A default security list with default security rules may be provided for a VCN. DHCP options configured for a VCN provide configuration information that is automatically provided to instances in a VCN when they boot up.

[0144] In certain embodiments, configuration information for a VCN is determined and stored by a VCN control plane. The configuration information for a VCN may include, for example, information about an address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, the VCN distribution service publishes the configuration information stored by the VCN control plane, or a portion thereof, to the NVD. The distributed information may be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to and from compute instances in the VCN.

[0145] In a particular embodiment, the creation of VCNs and subnets is handled by a VCN control plane (CP), and the launch of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instances, and then calls the VCN control plane to create VNICs and attach the VNICs to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions.

[0146] A customer may create one or more VCNs using resources hosted by CSPI. Compute instances deployed on the customer VCN may communicate with different endpoints. These endpoints may include endpoints hosted by CSPI and endpoints external to CSPI.

[0147] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 11, 12, 13, 14, and 15 and described below. Figure 11 is a high-level diagram of a distributed environment 1100 showing an overlay or customer VCN hosted by CSPI, according to certain embodiments. The distributed environment shown in Figure 11 includes multiple components in an overlay network. The distributed environment 1100 shown in Figure 11 is only an example and is not intended to unduly limit the scope of the embodiments described in the claims. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 11 may have more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may have a different configuration or arrangement of systems.

[0148] As shown in the example of FIG. 11, a distributed environment 1100 includes a CSPI 1101 that provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, the CSPI 1101 provides IaaS services to subscribing customers. Data centers in the CSPI 1101 may be organized into one or more regions. An example of a region, “Region US” 1102, is shown in FIG. 11. A customer configures a customer VCN 1104 for the region 1102. A customer may deploy various compute instances on the VCN 1104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0149] In the embodiment shown in FIG. 11, customer VCN 1104 includes two subnets, namely "Subnet-1" and "Subnet-2," each with its own CIDR IP address range. In FIG. 11, the overlay IP address range of Subnet-1 is 10.0 / 16, and the address range of Subnet-2 is 10.1 / 16. VCN virtual router 1105 represents a logical gateway of the VCN that enables communication between the subnets of VCN 1104 and with other endpoints outside the VCN. VCN VR 1105 is configured to route traffic between VNICs in VCN 1104 and the gateway associated with VCN 1104. VCN VR 1105 provides a port to each subnet of VCN 1104. For example, VR 1105 may provide a port with IP address 10.0.0.1 to Subnet-1 and a port with IP address 10.1.0.1 to Subnet-2.

[0150] Multiple compute instances may be deployed on each subnet, and the compute instances may be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines in CSPI1101. The compute instances join the subnet through a VNIC associated with the compute instance. For example, as shown in FIG. 11, compute instance C1 is part of subnet-1 through a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet-1 through a VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and a MAC address through its associated VNIC. For example, in FIG. 11, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, and compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet-1, including compute instances C1 and C2, has a default route to VCN VR1105 using IP address 10.0.0.1, which is the IP address of a port in VCN VR1105 in Subnet-1.

[0151] Multiple compute instances, including virtual machine instances and / or bare metal instances, may be deployed on subnet-2. For example, as shown in FIG. 11, compute instances D1 and D2 are part of subnet-2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 11, compute instance D1 has an overlay IP address 10.1.0.2 and a MAC address MM1, and compute instance D2 has a private overlay IP address 10.1.0.3 and a MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR1105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR1105 in subnet-2.

[0152] VCN A 1104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic among subnets in the VCN.

[0153] A particular compute instance deployed on VCN 1104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 1200 and endpoints external to CSPI 1200. Endpoints hosted by CSPI 1101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints on a different subnet but in the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in different VCNs in the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 1106 or 1110, communication between a compute instance in subnet-1 and an endpoint in a service network 1110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 1108). Compute instances in a subnet hosted by CSPI 1101 may also communicate with endpoints not hosted by CSPI 1101 (i.e., external to CSPI 1101). These external endpoints include endpoints in a customer's on-premise network 1116, endpoints in other remote cloud-hosted networks 1118, public endpoints 1114 accessible via a public network such as the Internet, and other endpoints.

[0154] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2 in subnet-1. For a packet sent from the source compute instance whose destination is another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as necessary, and forwarding / routing the packet to the next hop to facilitate communication of the packet to its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then executes and forwards the packet to the destination compute instance.

[0155] When a packet is communicated from a compute instance in a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VRs. For example, if compute instance C1 in subnet-1 in FIG. 11 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR1 105 using the VCN VR's default route or port 10.0.0.1. VCN VR1 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0156] When a packet is communicated from a compute instance in VCN 1104 to an endpoint outside VCN 1104, the communication is facilitated by a VNIC associated with the source compute instance, VCN VR 1105, and a gateway associated with VCN 1104. One or more types of gateways may be associated with VCN 1104. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. A variety of different types of gateways may be configured for a VCN to facilitate different types of communications with different types of endpoints. Depending on the gateway, the communications may be over a public network (e.g., the Internet) or over a private network. A variety of communication protocols may be used for these communications.

[0157] For example, compute instance C1 may want to communicate with an endpoint outside of VCN 1104. The packet may first be processed by a VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside of Cl's subnet-1. The VNIC associated with C1 may forward the packet to VCN VR 1105 of VCN 1104. VCN VR 1105 may then process the packet and, as part of that processing, determine a particular gateway associated with VCN 1104 as the packet's next hop based on the packet's destination. VCN VR 1105 may then forward the packet to this particular identified gateway. For example, if the destination is an endpoint within the customer's operating premises network, the packet may be forwarded by VCN VR 1105 to a dynamic routing gateway (DRG) gateway 1122 configured for VCN 1104. The packet may then be forwarded from the gateway to the next hop to facilitate communication of the packet to its intended final destination.

[0158] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 11 and described below. As shown in the embodiment of FIG. 11, a dynamic routing gateway (DRG) 1122 may be added or associated with the customer VCN 1104 to provide a path for private network traffic communication between the customer VCN 1104 and another endpoint. The other endpoint may be the customer's on-premise network 1116, a VCN 1108 in a different region of CSPI 1101, or another remote cloud network 1118 not hosted by CSPI 1101. The customer on-premise network 1116 may be a customer network or customer data center built using the customer's resources. Access to the customer on-premise network 1116 is generally highly restricted. For a customer that has both a customer on-premise network 1116 and one or more VCNs 1104 deployed or hosted in the cloud by CSPI 1101, the customer may want their on-premise network 1116 and their cloud-based VCN 1104 to be able to communicate with each other. This allows a customer to build an extended hybrid environment that includes their own VCN 1104 hosted by CSPI 1101 and their on-premise network 1116. DRG 1122 enables this communication. To enable such communication, a communication channel 1124 is set up. One endpoint of the channel is in the customer on-premise network 1116 and the other endpoint is in CSPI 1101 and connected to the customer VCN 1104. The communication channel 1124 can be over a public communication network such as the Internet, or a private communication network. A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, Oracle's FastConnect technology using a private network instead of a public network, etc.A device or equipment in the customer on-premise network 1116 that forms one endpoint of the communication channel 1124 is called customer premises equipment (CPE), such as CPE 1126 shown in Figure 11. On the CSPI 1101 side, the endpoint may be a host machine running a DRG 1122.

[0159] In certain embodiments, remote peering connections (RPCs) can be added to a DRG, allowing a customer to peer one VCN with another VCN in a different region. Using such RPCs, a customer VCN 1104 can connect to a VCN 1108 in another region using a DRG 1122. The DRG 1122 can also be used to communicate with other remote cloud networks 1118 not hosted by CSPI 1101, such as the Microsoft® Azure cloud, the Amazon® AWS cloud, etc.

[0160] As shown in Figure 11, an Internet Gateway (IGW) 1120 may be configured for a customer VCN 1104 that allows compute instances on the VCN 1104 to communicate with public endpoints 1114 accessible over a public network, such as the Internet. The IGW 1120 is a gateway that connects a VCN to a public network, such as the Internet. The IGW 1120 allows public subnets in a VCN, such as VCN 1104 (resources in the public subnet have public overlay IP addresses) to directly access public endpoints 1112 over a public network 1114, such as the Internet. The IGW 1120 can be used to initiate connections from subnets in VCN 1104 or from the Internet.

[0161] A network address translation (NAT) gateway 1128 may be configured for a customer's VCN 1104 to allow cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet and to do so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This allows private subnets in the VCN, such as private subnet-1 in VCN 1104, to privately access public endpoints on the Internet. With a NAT gateway, connections can only be initiated from the private subnet to the public Internet, not from the Internet to the private subnet.

[0162] In certain embodiments, a service gateway (SGW) 1126 may be configured for the customer VCN 1104 to provide a path for private network traffic between the VCN 1104 and service endpoints supported in the service network 1110. In certain embodiments, the service network 1110 may be provided by a CSP and may provide a variety of services. One example of such a service network is Oracle's Services Network, which provides a variety of services that customers can use. For example, a compute instance (e.g., a database system) in a private subnet of the customer VCN 1104 can back up data to a service endpoint (e.g., Object Storage) without needing a public IP address or access to the Internet. In certain embodiments, a VCN can have only one SGW, and connections can only be initiated from subnets in the VCN, not from the service network 1110. When a VCN is peered with another VCN, resources in this other VCN typically do not have access to the SGW. Also, resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can use a service gateway configured for that VCN.

[0163] In a particular implementation, the SGW 1126 uses the concept of a service classless inter-domain routing (CIDR) label, which is a string that represents all the regional public IP address ranges for a service or group of services of interest. A customer uses the service CIDR label when configuring the SGW and associated route rules to control traffic to the service. A customer can optionally utilize the service CIDR label when configuring security rules without having to adjust the security rules if the service's public IP address changes in the future.

[0164] A local peering gateway (LPG) 1132 is a gateway that can be added to a customer VCN 1104 to allow the VCN 1104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic traversing a public network such as the Internet or routing the traffic through the customer's on-premises network 1116. In a preferred embodiment, a VCN has a separate LPG for each peering that the VCN establishes. Local peering or VCN peering is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0165] A service provider, such as a provider of services in service network 1110, may provide access to services using different access models. According to a public access model, a service may be exposed as a public endpoint publicly accessible by compute instances in the customer VCN over a public network such as the Internet, and / or may be privately accessible through SGW 1126. According to a specific private access model, a service is made accessible as a private IP endpoint in a private subnet in the customer's VCN. This is called private endpoint (PE) access and allows a service provider to expose its services as instances in the customer's private network. A private endpoint resource represents a service in a customer's VCN. Each PE appears as a VNIC (called a PE-VNIC, with one or more private IPs) in a subnet selected by the customer in the customer's VCN. Thus, the PE provides a way to present a service in a private customer VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, all characteristics associated with the VNIC, such as routing rules, security lists, etc., are available to the PE VNIC.

[0166] Service providers can enable access through the PE by registering their own services. Providers can associate policies with services that limit the visibility of the service to customer tenancies. Providers can register multiple services under one virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) representing the same service.

[0167] Then, compute instances in the private subnet can access the service using the private IP address of the PE VNIC or the service DNS name. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 1130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 1110) that serves as an ingress / egress point for all traffic to and from customer subnet private endpoints. The PAGW 1130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in one VCN. The provider can represent the services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears not to be attached to the customer's instance, but to the service the customer wants to interact with. Traffic destined for the private endpoint is routed to the service through the PAGW 1130. These are called customer-to-service private connections (C2S connections).

[0168] The PE concept can also be used to extend private access of services to customer on-premises networks and data centers by allowing traffic to pass through FastConnect / IPSec links and private endpoints in the customer VCN, and to extend private access of services to customer peered VCNs by allowing traffic to pass between LPG1132 and a PE in the customer VCN.

[0169] Customers can control routing within a VCN at the subnet level, so they can specify which subnets in a customer's VCN, such as VCN 1104, use each gateway. The VCN's route tables are used to determine whether traffic is allowed to exit the VCN through a particular gateway. For example, in a particular case, the route table for a public subnet in customer VCN 1104 may send non-local traffic through IGW 1120. The route table for a private subnet in the same customer VCN 1104 may send traffic destined for CSP services through SGW 1126. All remaining traffic may be sent through NAT gateway 1128. The route tables control only traffic that exits the VCN.

[0170] Security lists associated with a VCN are used to control traffic coming into the VCN through a gateway via an inbound connection. All resources within a subnet use the same route table and security lists. Security lists may be used to control the specific types of traffic that are allowed to flow to and from instances within a subnet of a VCN. Security list rules may include ingress (inbound) and egress (outbound) rules. For example, an ingress rule may specify an allowed source address range, and an egress rule may specify an allowed destination address range. Security rules may specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In certain implementations, the operating system of the instance may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without an explicit security list rule for the response traffic) or stateless.

[0171] Access from a customer VCN (i.e., by resources or compute instances deployed on VCN 1104) may be categorized as public access, private access, or dedicated access. Public access refers to an access model that uses a public IP address or NAT to access a public endpoint. Private access allows customer workloads (e.g., resources in a private subnet) in VCN 1104 with private IP addresses to access a service without traversing a public network such as the Internet. In certain embodiments, CSPI 1101 allows customer VCN workloads with private IP addresses to access (public service endpoints of) a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoints of the service that reside outside the customer's private network.

[0172] In addition, CSPI may provide dedicated public access using technologies such as FastConnect public peering, where customer on-premises instances can access one or more services in the customer VCN using a FastConnect connection and without traversing a public network such as the Internet. CSPI may also provide dedicated private access using FastConnect private peering, where customer on-premises instances with private IP addresses can access customer VCN workloads using a FastConnect connection. FastConnect is an alternative network connectivity to connect a customer on-premises network to CSPI and its services using the public Internet. FastConnect provides an easy, flexible, and economical way to create dedicated private connections with higher bandwidth options and a more reliable and consistent networking experience compared to Internet-based connections.

[0173] FIG. 11 and the accompanying description above describe various virtualized components in an exemplary virtual network. As mentioned above, a virtual network is built on an underlying physical or backbone network. FIG. 12 illustrates a simplified architecture diagram of physical components in a physical network in CSPI 1200 that provides the basis for a virtual network, according to a particular embodiment. As illustrated, CSPI 1200 provides a distributed environment including components and resources (e.g., compute resources, memory resources, and networking resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services to which the customer subscribes, a subset of the resources (e.g., compute resources, memory resources, and networking resources) of CSPI 1200 is provisioned to the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network using the physical compute resources, memory resources, and networking resources provided by CSPI 1200. As mentioned above, these customer networks are called Virtual Cloud Networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. The compute instances can take the form of virtual machines, bare metal instances, etc. CSPI1200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0174] In the exemplary embodiment shown in FIG. 12, the physical components of CSPI 1200 include one or more physical host machines or servers (e.g., 1202, 1206, 1208), network virtualization devices (NVDs) (e.g., 1210, 1212), top-of-rack (TOR) switches (e.g., 1214, 1216), a physical network (e.g., 1218), and switches within physical network 1218. The physical host machines or servers may host and execute various compute instances that participate in one or more subnets of a VCN. The compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 11 may be hosted by the physical host machines shown in FIG. 12. The virtual machine compute instances in a VCN may be executed by one host machine or by multiple different host machines. The physical host machines may also host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 11 may be executed by the NVDs shown in FIG. 12. The gateway shown in FIG. 11 may be implemented by a host machine and / or NVD shown in FIG.

[0175] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances may be created, executed, and managed on a host machine by a hypervisor on that host machine. The hypervisor on a host machine enables the physical computing resources of the host machine (e.g., compute resources, memory resources, and networking resources) to be shared among the various compute instances executed by the host machine.

[0176] For example, as shown in FIG. 12, host machines 1202 and 1208 run hypervisors 1260 and 1266, respectively. These hypervisors may be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that sits on top of the host machine's operating system (OS), which runs on the host machine's hardware processors. A hypervisor provides a virtualized environment by allowing the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources, etc.) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 12, hypervisor 1260 may sit on the OS of host machine 1202 and allow the host machine's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) executed by host machine 1202. A virtual machine may have its own operating system (called a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor allows multiple operating systems to run side by side with each other while sharing the same computing resources of the host machine. The host machines shown in FIG. 12 may have the same type of hypervisor or different types of hypervisors.

[0177] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 12, compute instance 1268 on host machine 1202 and compute instance 1274 on host machine 1208 are examples of virtual machine instances. Host machine 1206 is an example of a bare metal instance provided to a customer.

[0178] In certain instances, an entire host machine may be provisioned for a single customer, and all of the one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine belong to that same customer. In other instances, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be members of different VCNs for different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting that bare metal instance, and the host machine is not shared with other customers or tenants.

[0179] As described above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. When a compute instance is created, a VNIC is associated with the compute instance. In a particular embodiment, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in FIG. 12, the host machine 1202 executes a virtual machine compute instance 1268 associated with a VNIC 1276, which is executed by an NVD 1210 connected to the host machine 1202. As another example, a bare metal instance 1272 hosted by the host machine 1206 is associated with a VNIC 1280 executed by an NVD 1212 connected to the host machine 1206. As yet another example, the VNIC 1284 is associated with a compute instance 1274 executed by the host machine 1208 , and the VNIC 1284 is executed by the NVD 1212 connected to the host machine 1208 .

[0180] For a compute instance hosted by a host machine, the NVD connected to that host machine also runs a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 12, NVD 1210 runs VCN VR 1277 corresponding to the VCN of which compute instance 1268 is a member. NVD 1212 may also run one or more VCN VRs 1283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 1206 and 1208.

[0181] A host machine may include one or more network interface cards (NICs) that allow the host machine to be connected to other devices. The NICs on a host machine may provide one or more ports (or interfaces) that allow the host machine to be communicatively connected to other devices. For example, one or more ports (or interfaces) provided on a host machine and on an NVD may be used to connect the host machine to the NVD. A host machine may also be connected to other devices, such as another host machine.

[0182] 12, host machine 1202 is connected to NVD 1210 using link 1220 extending between port 1234 provided by NIC 1232 of host machine 1202 and port 1236 of NVD 1210. Host machine 1206 is connected to NVD 1212 using link 1224 extending between port 1246 provided by NIC 1244 of host machine 1206 and port 1248 of NVD 1212. Host machine 1208 is connected to NVD 1212 using link 1226 extending between port 1252 provided by NIC 1250 of host machine 1208 and port 1254 of NVD 1212.

[0183] The NVDs are in turn connected via communication links to top-of-rack (TOR) switches that are connected to a physical network 1218 (also called a switch fabric). In a particular embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in FIG. 12, NVDs 1210 and 1212 are connected to TOR switches 1214 and 1216, respectively, using links 1228 and 1230. In a particular embodiment, links 1220, 1224, 1226, 1228, and 1230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0184] The physical network 1218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 1218 can be a multi-tier network. In a particular implementation, the physical network 1218 is a multi-tier Clos network of switches, with the TOR switches 1214 and 1216 representing leaf-level nodes of the multi-tier and multi-node physical switching network 1218. Different Clos network configurations are possible, including but not limited to 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and generally "n"-tier networks. An example of a Clos network is shown in FIG. 15 and described below.

[0185] A variety of different connection configurations are possible between host machines and NVDs, including one-to-one, many-to-one, and one-to-many. In a one-to-one implementation, each host machine is connected to its own separate NVD. For example, in FIG. 12, host machine 1202 is connected to NVD 1210 via NIC 1232 of host machine 1202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in FIG. 12, host machines 1206 and 1208 are connected to the same NVD 1212 via NICs 1244 and 1250, respectively.

[0186] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 13 shows an example in CSPI 1300 where a host machine is connected to multiple NVDs. As shown in Figure 13, a host machine 1302 includes a network interface card (NIC) 1304 including multiple ports 1306 and 1308. The host machine 1300 is connected to a first NVD 1310 via port 1306 and link 1320, and is connected to a second NVD 1312 via port 1308 and link 1322. The ports 1306 and 1308 may be Ethernet ports, and the links 1320 and 1322 between the host machine 1302 and the NVDs 1310 and 1312 may be Ethernet links. The NVD 1310 is then connected to a first TOR switch 1314, and the NVD 1312 is connected to a second TOR switch 1316. The links between the NVDs 1310 and 1312 and the TOR switches 1314 and 1316 may be Ethernet links. The TOR switches 1314 and 1316 represent layer-0 switching devices in a multi-layer physical network 1318.

[0187] 13 provides two separate physical network paths from the physical switch network 1318 to the host machine 1302: a first path traversing the TOR switch 1314 to the NVD 1310 and the host machine 1302, and a second path traversing the TOR switch 1316 to the NVD 1312 and the host machine 1302. These separate paths provide improved availability (called high availability) of the host machine 1302. If there is a problem with one path (e.g., a link on one path goes down) or there is a problem with a device (e.g., a particular NVD is not functioning), the other path may be used for communication to and from the host machine 1302.

[0188] In the configuration shown in Figure 13, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that allow the host machine to be connected to multiple NVDs.

[0189] Referring again to Figure 12, an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0190] The NVD may be implemented in a variety of different forms. For example, in a particular embodiment, the NVD is implemented as an interface card, called a smart NIC or intelligent NIC, with an embedded processor. The smart NIC is a separate device from the NIC on the host machine. In FIG. 12, the NVD 1210 may be implemented as a smart NIC connected to the host machine 1202, and the NVD 1212 may be implemented as a smart NIC connected to the host machines 1206 and 1208.

[0191] However, smart NICs are only one implementation of the NVD. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated in or performed by one or more host machines, one or more TOR switches, and other components of CSPI1200. For example, the NVD may be embodied in a host machine and the functions performed by the NVD may be performed by the host machine. As another example, the NVD may be part of a TOR switch, or a TOR switch may be configured to perform the functions performed by the NVD that enable the TOR switch to perform various complex packet transformations used in the public cloud. A TOR that performs the functions of the NVD may be referred to as a smart TOR. In yet other implementations where customers are provided with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0192] In certain embodiments, such as when implemented as a smart NIC as shown in FIG. 12, an NVD may include multiple physical ports that allow the NVD to be connected to one or more host machines and one or more TOR switches. Ports on an NVD may be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). A host-facing port of an NVD is a port used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 12 include port 1236 on NVD 1210, and ports 1248 and 1254 on NVD 1212. A network-facing port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 12 include port 1256 on NVD 1210, and port 1258 on NVD 1212. 12, the NVD 1210 is connected to the TOR switch 1214 using a link 1228 that extends from a port 1256 of the NVD 1210 to the TOR switch 1214. Similarly, the NVD 1212 is connected to the TOR switch 1216 using a link 1230 that extends from a port 1258 of the NVD 1212 to the TOR switch 1216.

[0193] The NVD may receive packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-facing port, perform necessary packet processing, and then forward the packets and frames to a TOR switch via the NVD's network-facing port. The NVD may receive packets and frames from a TOR switch via the NVD's network-facing port, and perform necessary packet processing, and then forward the packets and frames to a host machine via the NVD's host-facing port.

[0194] In certain embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (called a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as one logical link. All physical links in a given LAG may operate at the same speed and in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG goes down, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link achieves a higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as one logical port. Traffic can be load balanced across the multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be between the NVD and the TOR switch, between a host machine and the NVD, etc.

[0195] The NVD implements or executes network virtualization functions. These functions are executed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances in the VCN, etc. In a particular embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline to process the packet and determine how to forward or route the packet. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with the overlay network, such as implementing VNICs associated with compute instances in the VCN, implementing virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, implementing specific gateways (e.g., local peering gateways), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0196] In a particular embodiment, the packet processing data path in the NVD may include multiple packet pipelines, each of which may consist of a series of packet transformation stages. In a particular implementation, when a packet is received, it is parsed and classified into a pipeline. The packet is then processed linearly from stage to stage until the packet is dropped or sent out through an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., validating headers, performing throttling, inserting new layer 2 headers, performing L4 firewalling, VCN encapsulation / decapsulation, etc.), so that new pipelines can be constructed by composing existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0197] The NVD may perform both control plane and data plane functions corresponding to the control and data planes of the VCN. The control plane functions include functions used to configure the network (e.g., setting up routes and route tables, configuring VNICs, etc.) that control how data is forwarded. In certain embodiments, a VCN control plane is provided that centrally computes all overlay-to-substrate mappings and exposes them to the NVD as well as to virtual network edge devices such as various gateways such as DRGs, SGWs, IGWs, etc. Firewall rules may also be exposed using the same mechanism. In certain embodiments, the NVD retrieves only the mappings that are relevant to that NVD. The data plane functions include functions for actually routing / forwarding packets based on the configurations set up using the control plane. The VCN data plane is achieved by encapsulating customer network packets before they traverse the backbone network. The encapsulation / decapsulation functions are achieved on the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machines and perform network virtualization functions.

[0198] As described above, the NVD performs various virtualization functions including VNICs and VCN VRs. The NVD may perform VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 12, the NVD 1210 performs functions for a VNIC 1276 associated with a compute instance 1268 hosted by a host machine 1202 connected to the NVD 1210. As another example, the NVD 1212 performs a VNIC 1280 associated with a bare metal compute instance 1272 hosted by a host machine 1206 and a VNIC 1284 associated with a compute instance 1274 hosted by a host machine 1208. The host machines may host compute instances belonging to different VCNs that belong to different customers, and the NVDs connected to the host machines may perform VNICs (i.e., perform functions related to the VNICs) corresponding to the compute instances.

[0199] NVDs also run VCN virtual routers corresponding to the VCNs of compute instances. For example, in the embodiment shown in FIG. 12, NVD 1210 runs VCN VR 1277 corresponding to the VCN to which compute instance 1268 belongs. NVD 1212 runs one or more VCN VRs 1283 corresponding to one or more VCNs to which compute instances hosted by host machines 1206 and 1208 belong. In particular embodiments, the VCN VR corresponding to that VCN is run by all NVDs connected to a host machine that hosts at least one compute instance that belongs to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to those different VCNs.

[0200] In addition to the VNICs and VCN VRs, the NVD may include one or more hardware components that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped as "packet processing components" shown in FIG. 12. For example, the NVD 1210 includes a packet processing component 1286, and the NVD 1212 includes a packet processing component 1288. For example, the packet processing component of the NVD may include a packet processor configured to monitor all packets received by the NVD and communicated using the NVD by interacting with the ports and hardware interfaces of the NVD and store network information. The network information may include, for example, network flow information identifying different network flows processed by the NVD, and per-flow information (e.g., per-flow statistics). In certain embodiments, the network flow information may be stored on a per-VNIC basis. The packet processor may perform per-packet operations as well as realize stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly the state and health of other components connected to the NVD.

[0201] FIG. 11 illustrates components of an exemplary virtual or overlay network, including a VCN, a subnet in the VCN, compute instances deployed on the subnet, VNICs associated with the compute instances, VRs of the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 11 may be executed or hosted by one or more of the physical components illustrated in FIG. 12. For example, a compute instance in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 12. For a compute instance hosted by a host machine, the VNICs associated with that compute instance are typically executed by an NVD connected to that host machine (i.e., the VNIC functionality is provided by an NVD connected to that host machine). The VCN VR functionality of the VCN is executed by all NVDs connected to the host machines that host or execute the compute instances that are part of that VCN. The gateways associated with the VCN may be executed by one or more different types of NVDs. For example, certain gateways may be executed by smart NICs, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0202] As mentioned above, the compute instances in a customer VCN may communicate with a variety of different endpoints, which may be in the same subnet as the source compute instance, in a different subnet but in the same VCN as the source compute instance, or may have endpoints outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0203] In the case of communication between two compute instances on the same subnet in a VCN, the communication is facilitated using VNICs associated with the source compute instance and the destination compute instance. The source compute instance and the destination compute instance may be hosted by the same host machine or may be hosted by different host machines. A packet sent from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to the host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the destination endpoint of the packet is in the same subnet, running the VNIC associated with the source compute instance forwards the packet to the NVD running the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be running on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs). The VNIC may use the routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0204] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet outgoing from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also referred to as executing the VNIC). The function executed by the VNIC may include noting the VLAN tag on the packet. Because the destination of the packet is outside the subnet, a VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or may run on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0205] If the destination of the packet is outside the VCN of the source compute instance, the packet leaving the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which may result in the packet being forwarded to an NVD running an appropriate gateway associated with the VCN. For example, if the destination is an endpoint in the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be run on the same NVD as the NVD running the VNIC associated with the source compute instance, or may be run by a different NVD. The gateway may be run by the NVD, which may be a smart NIC, a host machine, or other NVD implementation. The packet is then processed by the gateway and forwarded to the next hop that facilitates communication of the packet to the intended destination endpoint. 12, a packet outgoing from compute instance 1268 may be communicated from host machine 1202 to NVD 1210 over link 1220 (using NIC 1232). On NVD 1210, VNIC 1276 is invoked because VNIC 1276 is the VNIC associated with source compute instance 1268. VNIC 1276 is configured to examine encapsulation information in the packet, determine a next hop for forwarding the packet to facilitate communication of the packet to an intended destination endpoint, and then forward the packet to the determined next hop.

[0206] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI1200 and endpoints external to CSPI1200. Endpoints hosted by CSPI1200 may include instances in the same VCN or in other VCNs, which may be the customer's VCN or VCNs not belonging to the customer. Communications between endpoints hosted by CSPI1200 may be performed over physical network 1218. Compute instances may also communicate with endpoints not hosted by CSPI1200 or external to CSPI1200. Examples of these endpoints include endpoints in the customer's on-premise network or datacenter, or public endpoints accessible over a public network such as the Internet. Communications with endpoints external to CSPI1200 may be performed over a public network (e.g., the Internet) (not shown in FIG. 12) or a private network (not shown in FIG. 12) using a variety of communication protocols.

[0207] The architecture of CSPI1200 shown in FIG. 12 is an example only and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI1200 may have more or fewer systems or components than those shown in FIG. 12, may combine two or more systems, or may have a different configuration or arrangement of systems. The systems, subsystems, and other components shown in FIG. 12 may be implemented with software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

[0208] FIG. 14 illustrates connectivity between host machines and NVDs to provide I / O virtualization to support multitenancy, according to certain embodiments. As shown in FIG. 14, host machine 1402 runs hypervisor 1404 that provides a virtualized environment. Host machine 1402 runs two virtual machine instances: VM1 1406 belonging to customer / tenant #1 and VM2 1408 belonging to customer / tenant #2. Host machine 1402 includes a physical NIC 1410 that is connected to NVD 1412 via link 1414. Each of the compute instances is attached to a VNIC that is executed by NVD 1412. In the embodiment of FIG. 14, VM1 1406 is attached to VNIC-VM1 1420 and VM2 1408 is attached to VNIC-VM2 1422.

[0209] As shown in Figure 14, NIC 1410 includes two logical NICs: logical NIC A 1416 and logical NIC B 1418. Each virtual machine is attached to and configured to operate with its own logical NIC. For example, VM1 1406 is attached to logical NIC A 1416 and VM2 1408 is attached to logical NIC B 1418. Although host machine 1402 contains only one physical NIC 1410 shared by multiple tenants, the logical NICs allow each tenant's virtual machine to believe it has its own host machine and NIC.

[0210] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 1416 for tenant #1 is assigned a particular VLAN ID, and logical NIC B 1418 for tenant #2 is assigned a different VLAN ID. When a packet is communicated from VM1 1406, a tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 1402 to NVD 1412 over link 1414. Similarly, when a packet is communicated from VM2 1408, a tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 1402 to NVD 1412 over link 1414. Thus, a packet 1424 communicated from host machine 1402 to NVD 1412 has an associated tag 1426 that identifies the particular tenant and associated VM. When a packet 1424 is received on the NVD from host machine 1402, the tag 1426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 1420 or VNIC-VM2 1422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 14 allows each tenant's compute instance to believe that it has its own host machine and NIC. The setup shown in Figure 14 provides I / O virtualization to support multi-tenancy.

[0211] FIG. 15 illustrates a simplified block diagram of a physical network 1500 according to a particular embodiment. The embodiment illustrated in FIG. 15 is constructed as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers can be 2, 3, 4, 5, etc. The embodiment illustrated in FIG. 15 is a three-layer network including layers 1, 2, and 3. A TOR switch 1504 represents a layer-0 switch in a Clos network. One or more NVDs are connected to a TOR switch. A layer-0 switch is also referred to as an edge device of a physical network. A layer-0 switch is connected to a layer-1 switch, also referred to as a leaf switch. In the embodiment illustrated in FIG. 15, a set of "n" layer-0 TOR switches are connected to a set of "n" layer-1 switches, together forming a pod. Each layer-0 switch in a pod is interconnected to all layer-1 switches within the pod, but there is no switch connectivity between pods. In a particular implementation, the two pods are called blocks. Each block is served by or connected to a set of "n" layer-2 switches (sometimes called spine switches). There may be several blocks in the physical network topology. The layer-2 switches are then connected to "n" layer-3 switches (sometimes called super-spine switches). Communication of packets over the physical network 1500 is typically performed using one or more layer-3 communication protocols. Typically, all layers of the physical network except the TOR layer are n-way redundant, thus enabling high availability. Policies may be specified for the pods and blocks that control the mutual visibility of switches in the physical network to enable scaling of the physical network.

[0212] A feature of Clos networks is that the maximum hop count to reach from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is fixed. For example, in a three-tier Clos network, a maximum of seven hops are required for a packet to reach from one NVD to another, and the source NVD and the target NVD are connected to the leaf layers of the Clos network. Similarly, in a four-tier Clos network, a maximum of nine hops are required for a packet to reach from one NVD to another, and the source NVD and the target NVD are connected to the leaf layers of the Clos network. Thus, the Clos network architecture keeps the latency constant throughout the network, which is important for intra-datacenter and inter-datacenter communications. Clos topologies scale horizontally and are cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches (e.g., more leaf and spine switches) to various tiers and by increasing the number of links between switches in adjacent tiers.

[0213] In a particular embodiment, each resource in the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or through an API. An exemplary syntax for a CID is: ocid1.<RESOURCE TYPE> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> In the formula: "ocid1" is a literal string indicating the version of the CID.

[0214] "resource type" is the type of resource (for example, instance, volume, VCN, subnet, user, group, etc.).

[0215] "realm" is the realm in which the resource resides. Example values ​​are "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal cloud realm, etc. Each realm may have its own domain name.

[0216] "region" is the region the resource is in. This part may be blank if no region applies to the resource.

[0217] "Future use" is reserved for future use. The "unique ID" is the unique part of the ID. The format can vary depending on the type of resource or service.

[0218] Although specific embodiments of the present disclosure have been described, various modifications, alternatives, alternative configurations, and equivalents are also encompassed within the scope of the present disclosure. The embodiments of the present disclosure are not limited to operating in a particular specific data processing environment, but can freely operate in multiple data processing environments. In addition, although the embodiments of the present disclosure have been described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that the scope of the present disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the above-described embodiments may be used individually or jointly.

[0219] Furthermore, while embodiments of the present disclosure have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. Embodiments of the present disclosure may be implemented using only hardware, only software, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination. Thus, when a component or module is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0220] The specification and drawings are therefore to be regarded as illustrative rather than restrictive. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope of the appended claims. Thus, although specific embodiments of the disclosure have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be within the scope of the appended claims.

[0221] The use of the words "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be construed to include both the singular and the plural, unless otherwise stated herein or clearly contradicted by context. The words "comprising," "having," "including," and "containing" should be construed as open-ended (i.e., meaning "including but not limited to"), unless otherwise stated. The word "connected" should be construed as partly or wholly contained, attached, or coupled, even if there is something intervening. The recitation of ranges of values ​​herein is merely intended as a shorthand way of referring individually to each individual value contained within the range, unless otherwise stated herein, and each individual value is incorporated herein as if it were individually set forth herein. All methods described herein can be performed in any suitable order, unless otherwise stated herein or clearly contradicted by context. Any examples provided herein, or the use of exemplary language (e.g., "such as"), are intended merely to better illustrate embodiments of the disclosure and do not limit the scope of the disclosure unless specifically claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0222] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood within the context in which it is commonly used to indicate that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless specifically stated otherwise. Thus, such disjunctive language is generally not intended to, and should not, imply that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0223] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the present disclosure. Modifications of these preferred embodiments will become apparent to those skilled in the art upon reading the above description. Those skilled in the art should be able to adopt such modifications as appropriate, and the present disclosure may be carried out in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure, unless otherwise indicated herein.

[0224] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each individual reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0225] Although the above specification describes aspects of the disclosure with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.< / realm>

Claims

1. 1. A system comprising: a first virtualization engine configured to instantiate a first virtual machine, the first virtualization engine comprising: a first memory device configured to store a first key, the first key being a cryptographic key, the first virtualization engine further comprising: a first crypto processor configured to encrypt and / or decrypt data to and / or from the first virtual machine using the first key, the system further comprising: a second virtualization engine configured to instantiate a second virtual machine, the second virtualization engine comprising: a second memory device configured to store a second key, the second key being a cryptographic key, the second virtualization engine further comprising: a second crypto processor configured to encrypt and / or decrypt data to and / or from the second virtual machine using the second key; the first virtualization engine and the second virtualization engine are part of a device; A system, wherein a host of the device does not have access to the first key and the second key.

2. 2. The system of claim 1, wherein the first virtualization engine is configured to receive the first key from a key management service and store the first key in the first memory device.

3. 3. The system of claim 2, wherein the key management service provides the first key to the device without the host of the device having access to the first key for a customer.

4. The system of claim 2 , wherein the first virtualization engine provides credentials to the key management service to receive the first key.

5. The system of any one of claims 1 to 4, wherein the first memory device is a static random access memory device including a capacitor and a transistor.

6. The system of any one of claims 1 to 5, wherein the first virtualization engine and the second virtualization engine are each configured to instantiate only one virtual machine at a time.

7. The system of any one of claims 1 to 6, wherein the first virtual machine is part of a first virtual cloud network and the second virtual machine is part of a second virtual cloud network.

8. The system of any preceding claim, wherein the first cryptographic processor is configured to provide in-line data encryption for the first virtual machine.

9. The system of any preceding claim, wherein the first virtualization engine is configured to provide network routing of data packets for the first virtual machine.

10. The system of any one of claims 1 to 9, wherein the first key is the same as the second key.

11. The system of any one of claims 1 to 10, wherein the device is a network interface card.

12. The system of any one of claims 1 to 11, wherein the first key is periodically updated.

13. The system of any one of claims 1 to 12, wherein the first virtual machine and the second virtual machine are part of a subnetwork.

14. The system of any one of claims 1 to 12, wherein the first virtual machine and the second virtual machine are part of different virtual cloud networks.

15. The system of any one of claims 1 to 14, wherein the first key and the second key are refreshed simultaneously.

16. 1. A method comprising: instantiating a first virtual machine using a first virtualization engine; instantiating a second virtual machine using a second virtualization engine; storing a first key in a first memory device of the first virtualization engine; storing a second key in a second memory device of the second virtualization engine; and encrypting data of the first virtual machine using a first cryptographic processor and the first key, the first cryptographic processor being part of the first virtualization engine, the method further comprising: encrypting data of the second virtual machine using a second cryptographic processor and the second key; the second cryptographic processor is part of the second virtualization engine; the first virtualization engine and the second virtualization engine are part of a device; A method, wherein a host of the device does not have access to the first key or the second key.

17. The method of claim 16 , wherein the first key is different from the second key.

18. 18. The method of claim 16, wherein the first memory device is configured to store keys for only the first virtual machine.

19. A computer program for causing one or more processors to carry out the method according to any one of claims 16 to 18.