Virtual network distributed denial of service cleaner

Abstract

A novel overlay network DDoS mitigation system (ONDMS) for performing DDoS attack mitigation in a virtual network environment is described. Network traffic received by network resources in an overlay network is monitored. When a potential DDoS attack is detected, the ONDMS may initiate a protected mode for the network resource. This may involve creating one or more shadow VNIC for the network resource being protected. When in the protected mode, due to the one or more shadow VNIC, packets that would otherwise be received by the network resource being protected are changed to be redirected to one or more alternative destinations (e.g., a DDoS cleaner system within the ONDMS) that would otherwise be received by the network resource being protected. The one or more alternative destinations are configured to filter and analyze packets and take appropriate mitigation actions as needed. This protects the network resources being protected from potential DDoS attacks.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Non - Provisional Application No. 18 / 544,272, titled "VIRTUAL NETWORK DISTRIBUTED DENIAL - OF - SERVICE SCRUBBER", filed on December 18, 2023, which claims the benefit and priority of U.S. Provisional Application No. 63 / 434,887, titled "VIRTUAL NETWORK DISTRIBUTED DENIAL - OF - SERVICE SCRUBBER", filed on December 22, 2022. The entire content of this application is incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure generally relates to the security of computers and networks, and more particularly to techniques for preventing or mitigating distributed denial - of - service (DDoS) attacks in overlay networks. More specifically, a novel overlay network DDoS mitigation system (ONDMS) for performing DDoS attack mitigation in a virtual network environment is described. Background Art

[0004] Virtual networking enables cloud service providers (CSPs) to provide cloud services to subscribing customers. CSPs typically provide the infrastructure for forming an overlay network and providing cloud services to the CSP's customers. The infrastructure provided by CSPs can include various physical and logical resources, including various computing, memory, and networking resources (also referred to as network resources). Virtual networking provides the appearance of a customized private network for customers over the shared resources provided in the CSP's infrastructure. The foundation of virtual network products is the ability to provide customer isolation for shared resources. Resources such as network resources, link bandwidth, and computing resources are important for the successful delivery of cloud services and affect the user experience. Therefore, it is very important to carefully monitor and manage these resources.

[0005] In today's connected world, CSP infrastructure is vulnerable to malicious attackers. Distributed Denial of Service (DDoS) attacks are a common example of such malicious attacks. The source of the attack may be outside the CSP's infrastructure or may even be within the CSP's infrastructure. For example, one or more customers of the CSP may send a large volume of data (e.g., a large number of packets per second) to the same network resource destination (e.g., a virtual network interface card (VNIC), a network virtualization device implementing one or more VNICs (e.g., a SmartNIC)). This high rate of data reception may cause congestion on the NVD or on the link from the top-of-rack (TOR) switch to the NVD. This may in turn cause the computational resources (e.g., CPU) on the NVD to be exhausted, resulting in a failure of the NVD or a degradation in the performance of the NVD. This may have an adverse impact on the cloud services provided by the CSP to its customers. Therefore, protecting network resources from such DDoS attacks is crucial for the CSP. Summary of the Invention

[0006] The present disclosure generally relates to the security of computers and networks and, more particularly, to techniques for preventing or mitigating Distributed Denial of Service (DDoS) attacks in an overlay network. More specifically, a novel Overlay Network DDoS Mitigation System (ONDMS) for performing DDoS attack mitigation in a virtual network environment is described. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, etc.

[0007] The techniques described herein can be used in a cloud environment to prevent or mitigate DDoS attacks on cloud infrastructure and workloads deployed in the cloud. For example, a Cloud Service Provider (CSP) can provide an Overlay Network DDoS Mitigation System (ONDMS) in the CSP's cloud infrastructure, and the system can be used to prevent or mitigate DDoS attacks on the CSP's infrastructure.

[0008] In some embodiments, techniques are provided that include a method comprising: monitoring network traffic received by a network virtualization device (NVD) in a cloud service provider infrastructure, the NVD implementing a set of one or more virtual network interface cards (VNICs) associated with a set of one or more computing instances in one or more overlay networks provided by the cloud service provider infrastructure, the network traffic destined for at least one computing instance from the set of one or more computing instances; initiating, at least in part based on the monitoring, a protected mode for the NVD to protect the NVD from a potential distributed denial of service (DDoS) attack; and when the NVD is in the protected mode, causing one or more packets destined for the set of one or more computing instances to be redirected to a DDoS scrubber system rather than being sent to the NVD.

[0009] In yet another embodiment, initiating the protected mode for the NVD includes: determining the set of one or more VNICs implemented by the NVD, the set of one or more VNICs including a first VNIC associated with a first computing instance from the set of one or more computing instances, the first VNIC being associated with a first overlay address configured for the first computing instance, wherein the first overlay address is associated with a base address associated with the NVD; creating a set of one or more shadow VNICs for the set of one or more VNICs; associating the set of one or more shadow VNICs with the DDoS scrubber system; and publishing information indicating the set of one or more shadow VNICs to the one or more overlay networks provided by the cloud service provider infrastructure.

[0010] In yet another embodiment, causing the one or more packets destined for the set of one or more computing instances to be redirected to the DDoS scrubber system includes redirecting the one or more packets to the DDoS scrubber system due to the set of one or more shadow VNICs.

[0011] In yet another embodiment, the set of one or more VNICs includes a plurality of VNICs; and the set of one or more shadow VNICs includes a single shadow VNIC.

[0012] In yet another embodiment, the set of one or more VNICs includes a plurality of VNICs; and the set of one or more shadow VNICs includes a plurality of shadow VNICs, the plurality of shadow VNICs including a shadow VNIC corresponding to each VNIC in the plurality of VNICs.

[0013] In yet another embodiment, the set of one or more VNICs includes a plurality of VNICs; and the set of one or more shadow VNICs includes a plurality of shadow VNICs, wherein the number of shadow VNICs in the plurality of shadow VNICs is less than the number of VNICs in the plurality of VNICs.

[0014] In yet another embodiment, the DDoS scrubber system includes at least one host machine configured to implement at least one shadow VNIC from one or more shadow VNICs of the group or at least one NVD configured to implement at least one shadow VNIC from one or more shadow VNICs of the group.

[0015] In yet another embodiment, creating the one or more shadow VNICs for the one or more VNICs of the group includes: creating a first shadow VNIC corresponding to a first VNIC associated with a first computing instance and associating a first overlay address with the first shadow VNIC; and associating the one or more shadow VNICs of the group with the DDoS scrubber system includes associating the first shadow VNIC with a base address associated with the DDoS scrubber system.

[0016] In yet another embodiment, causing the one or more packets to be redirected to the DDoS scrubber system includes: for a first packet of the one or more packets, the first packet being sent to a first overlay address configured for a first computing instance; determining that for the first overlay address, the first packet is to be sent to a base address associated with the DDoS scrubber system; and sending the first packet to the DDoS scrubber system.

[0017] In yet another embodiment, the method further includes performing at least one action on at least one of the one or more packets by the DDoS scrubber system; wherein the performing includes discarding the one or more packets, throttling the one or more packets, or forwarding the one or more packets to an NVD.

[0018] In yet another embodiment, determining a potential distributed denial of service (DDoS) attack when the network traffic is above a predetermined threshold, the network traffic being above the predetermined threshold includes an average link utilization exceeding 80% for a continuous number of minutes or a link utilization burst reaching 100% or higher for a continuous number of minutes.

[0019] In yet another embodiment, the method further includes exiting a protected mode for the NVD; and after exiting, for any packet sent to a computing instance from one or more computing instances of the group, sending the packet to the NVD instead of redirecting the packet to the DDoS scrubber system.

[0020] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the one or more methods disclosed herein.

[0021] In certain embodiments, techniques are provided that include a method comprising: monitoring network traffic received by a first virtual network interface card (VNIC) associated with a first computing instance in an overlay network provided by cloud service provider infrastructure, the network traffic destined for the first computing instance; initiating, at least in part based on the monitoring, a protected mode for the first VNIC to protect the first VNIC from a potential distributed denial of service (DDoS) attack; and when the first VNIC is in the protected mode, causing one or more packets destined for the first computing instance to be redirected to a DDoS scrubber system rather than being sent to a first network virtualization device (NVD) implementing the first VNIC; wherein the first VNIC is associated with a first overlay address configured for the first computing instance, and the first overlay address is associated with a base address associated with the NVD implementing the first VNIC.

[0022] In certain embodiments, techniques are provided that include a method comprising: monitoring network traffic received by a plurality of network resources in one or more overlay networks provided by cloud service provider infrastructure, the network traffic destined for a first computing instance; initiating, at least in part based on the monitoring, a protected mode for a first network resource from among the plurality of network resources to protect the first network resource from a potential distributed denial of service (DDoS) attack, the first network resource being associated with the first computing instance; and when the first network resource is in the protected mode, causing one or more packets destined for the first computing instance to be redirected to a DDoS scrubber system rather than being sent to the first network resource.

[0023] In various embodiments, a non-transitory computer-readable medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors of a computer system to perform one or more methods disclosed herein.

[0024] In various embodiments, a computer program product includes a computer program / instructions that, when executed by a processor, cause the processor to perform any method disclosed herein.

[0025] The techniques described above and below can be implemented in many ways and in many contexts. Several example implementations and contexts are provided with reference to the following drawings, as described in more detail below. However, the following implementations and contexts are only a small portion of the many implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a high-level diagram showing a distributed environment of a virtual or overlay cloud network hosted by cloud service provider infrastructure in accordance with certain embodiments.

[0027] Figure 2 Depicted is a simplified architectural diagram of physical components in a physical network within a CSPI, according to certain embodiments.

[0028] Figure 3 An example arrangement within a CSPI where a host machine is connected to multiple network virtualization devices (NVDs) in accordance with certain embodiments is shown.

[0029] Figure 4 Depicted is connectivity between a host machine and an NVD for providing I / O virtualization to support multi-tenancy, according to certain embodiments.

[0030] Figure 5 Depicted is a simplified block diagram of a physical network provided by CSPI, in accordance with certain embodiments.

[0031] Figure 6 is a block diagram illustrating an example architecture for an Overlay Network DDoS Mitigation (ONDMS) in accordance with certain embodiments.

[0032] Figure 7A is a flow chart illustrating a process flow for entering a protected mode for a network resource according to certain embodiments.

[0033] Figure 7B is a flow chart illustrating a process flow for exiting protected mode for a network resource according to some embodiments.

[0034] Figure 8 is a flow chart illustrating the process flow for protected mode for an individual VNIC, according to certain embodiments.

[0035] Figure 9 is a flow chart illustrating a process flow for protected mode of a network virtualization device (NVD, eg, smartNIC), according to certain embodiments.

[0036] Figure 10 is a flow diagram illustrating packet redirection for an individual VNIC in protected mode according to certain embodiments.

[0037] Figure 11 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0038] Figure 12 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0039] Figure 13 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0040] Figure 14 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0041] Figure 15 is a block diagram illustrating an example computer system according to at least one embodiment.

[0042] Figure 16 is a flowchart illustrating a generalized processing flow for a protected mode for a network virtualization device (NVD, e.g., smartNIC) according to certain embodiments.

[0043] Figure 17 is a flowchart illustrating a generalized processing flow for a protected mode for an individual VNIC according to certain embodiments.

[0044] Figure 18 is a flowchart illustrating a generalized processing flow for a protected mode for network resources according to certain embodiments. DETAILED DESCRIPTION

[0045] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is clear that the various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0046] The present disclosure generally relates to the security of computers and networks, and more particularly to techniques for preventing or mitigating distributed denial of service (DDoS) attacks in overlay networks. More specifically, a novel overlay network DDoS mitigation system (ONDMS) for performing DDoS attack mitigation in a virtual network environment is described.

[0047] In certain embodiments, network traffic received by various networking resources (also referred to as network resources) in an overlay network provided by a cloud service provider (CSP) is monitored. The monitored networking resources can include logical resources such as virtual network interface cards (VNICs), physical resources such as network virtualization devices (NVDs), and the like. For example, packets destined for a particular compute instance (i.e., the particular compute instance is the destination of the packet) can be sent to an NVD that implements the VNIC associated with that particular compute instance. Packets sent to the VNIC can be monitored. As another example, an NVD can implement multiple VNICs associated with multiple compute instances. Packets destined for multiple compute instances are received by the NVD before being forwarded to the destination compute instance. The resource utilization and bandwidth of the NVD can be monitored.

[0048] Potential DDoS attack events are detected based on the monitoring of network resources. After detecting a potential DDoS attack event against a specific network resource in the overlay network, that specific resource is placed in a protected mode. When in the protected mode, packets that would otherwise be received by that specific network resource are instead redirected to one or more alternative destinations (e.g., a DDoS scrubber system), which are configured to filter and analyze the packets and take appropriate mitigation actions as needed. In some embodiments, one or more shadow VNICs are created for the network resources to be protected, and the shadow VNIC causes the network traffic to be redirected to the DDoS scrubber system. In this way, the network resources can be protected from potential DDoS attacks.

[0049] In a public cloud, virtual networks play an important role in resource utilization, security, flexibility, scalability, and cost savings. Although different customers (or tenants) perceive that they have their dedicated resources in the virtualized network, in fact these dedicated resources run on shared resources with sufficient security and isolation. Cloud service providers (CSPs) typically rely on well-behaved customers and their proper use of the equipment to avoid congestion. However, if some customers inadvertently or maliciously utilize more resources than those allocated or paid for by these customers, then this may affect the experience of other customers. For example, a customer sends or receives too much data over the network and causes congestion.

[0050] The network is a valuable and limited resource and can be constrained in two ways. One way is bandwidth (i.e., the number of gigabits per second (Gbps) on the network link). If some customers send or receive network traffic that is much higher than the bandwidth allocated to these customers, then these customers may prevent other customers from utilizing a fair share of the bandwidth available to other customers. This raises concerns about the limited bandwidth and available computing resources at the network virtualization device (NVD) (e.g., smartNIC) due to excessive traffic.

[0051] Another aspect is the packet rate (e.g., packets per second or PPS). Since each packet being transmitted or received needs to be processed, which consumes computing resources, excessive packets on the network consume excessive computing resources and may thus constrain the resources of other customers. For example, a customer may deliberately use some large instances to send a large number of packets to small instances, resulting in some target small instances being overloaded. On the other hand, in some cases, a customer may inadvertently misconfigure network features (e.g., virtual traffic access point (VTAP)). In other cases, an application may exhibit abnormal behavior. These situations may cause some instances of cloud resources to generate a large volume of traffic targeting a single instance or an instance that is too small (e.g., a VM).

[0052] From a virtual network perspective, there are two main distributed denial-of-service (DDoS) risks: transmission risk and reception risk. On the transmission side, important concerns are sending at a high packet rate, including control packets or other packets that require additional processing at a high rate, or a computing instance using too much bandwidth. These factors may cause CPU exhaustion on the NVD and bring about a noisy-neighbor problem to other instances sharing the same instance. Another concern is bandwidth. If the bandwidth of the sender is higher than that of the receiver, then this may potentially cause congestion on the receiver side, but it can be mitigated inside the computing instance by throttling to discard some packets.

[0053] On the reception side, one or more customers may send too much data or too many packets per second to the same destination VNIC or different destination VNICs on the same NVD (e.g., SmartNIC). An excessive data rate may cause congestion on the link between the top-of-rack (TOR) switch and the NVD, and an excessive packet rate may cause CPU exhaustion on the NVD. In the example shown, consider a scenario where traffic is sent by many sender instances. Each instance sends traffic within the allowed limits, but the aggregated traffic of all sender instances exceeds the limits of the receiver. Assume that each computing instance is allowed to send or receive 10 Gbps of traffic. Therefore, each of the ten computing instances sending 10 Gbps of traffic is compliant. However, when all ten computing instances send traffic to a single computing instance, it will result in traffic aggregation to 100 Gbps, which far exceeds the reception limits of the receiving computing instance. Additionally, assume that the receiving computing instance shares a server with other instances in a real application cluster (RAC). In this case, other instances may be affected by the aggregated traffic that exceeds the limits.

[0054] Some common DDoS-like problems are due to misconfigurations of features such as virtual traffic access points and improper application behavior, resulting in large volumes of traffic to small instances (e.g., virtual machines). Such problems may require the CSP to first identify the problems before the CSP can fix them. Thus, problem resolution is a time-consuming process. Sometimes, if the problems recur or become a real concern, then the CSP may block the customer or terminate the customer's account to prevent further problems. These methods are passive and reactive.

[0055] As indicated above, a novel overlay network DDoS mitigation system (ONDMS) for performing DDoS attack mitigation in a virtual network environment is described herein. The ONDMS monitors networked resources, including the usage and utilization of network resources. Based on the monitoring, the ONDMS determines whether and when to initiate a protected mode for a particular network resource. When the protected mode is initiated for a particular network device, packets that would otherwise be received by that particular network are instead redirected to one or more alternative destinations, such as a scrubber system that is part of the ONDMS. The scrubber system is configured to filter and analyze the packets and take appropriate mitigation actions as needed. In some embodiments, one or more shadow VNICs are created for the network resource to be protected, and the shadow VNICs cause network traffic to be redirected to the ONDMS. In this way, network resources can be protected from potential DDoS attacks.

[0056] The ONDMS described in this disclosure can be used to protect a variety of different network resources. Examples include one or more VNICs, NVDs, etc. For example, the network traffic received by a particular VNIC can be monitored. Based on the results of the monitoring or during the monitoring, the protected mode is initiated. In one aspect, based on the monitoring, when a threshold related to the traffic volume or a threshold related to the available bandwidth associated with the VNIC is reached or exceeded, it can be detected that the particular VNIC is under attack. After detecting that the particular VNIC is potentially under a DDoS attack, the protected mode is initiated for the particular VNIC. In this protected mode, a shadow VNIC is created for the particular VNIC. Information related to the shadow VNIC is published such that all network traffic (e.g., packets) received by the particular VNIC is instead redirected to the shadow VNIC. The shadow VNIC can be implemented by one or more systems or devices different from the system or device that implements the particular VNIC. In this way, the traffic is diverted to a device or system different from the device or system that implements the particular VNIC (referred to as the DDoS scrubber system). In this way, the particular VNIC, the device (e.g., NVD) that implements the particular VNIC, or the system (e.g., host machine) is protected from potential DDoS attacks.

[0057] A DDoS scrubber system that receives redirected packets is configured to analyze the redirected packets and take appropriate mitigation actions. Examples of mitigation actions can include: discarding all packets when in a protected mode; filtering packets and only allowing some filtered packets to reach a specific VNIC that is the intended destination of these packets; throttling packets such that the rate of packets sent to a specific VNIC is reduced, etc. In some embodiments, the DDoS scrubber system includes multiple machines (e.g., a team of host machines) that are configured to execute or implement multiple shadow VNICs.

[0058] As another example, the network traffic received by a specific NVD can be monitored. The NVD can execute or implement one or more VNICs. Based on the results of the monitoring or during the monitoring, a protected mode is initiated. In one aspect, based on the monitoring, when a threshold related to the traffic volume or a threshold related to the available bandwidth associated with the specific NVD is reached or exceeded, it can be detected that the specific NVD is under attack. After detecting that the specific NVD is potentially under a DDoS attack, a protected mode is initiated for the specific NVD. In this protected mode, one or more shadow VNICs are created for all VNICs implemented (or executed) by the specific NVD. In one embodiment, a single shadow VNIC is created for all VNICs executed by the specific NVD. In another embodiment, one or more shadow VNICs are created for the VNICs executed by the specific NVD, such as creating one shadow VNIC for each VNIC. Information related to the one or more shadow VNICs is published such that all network traffic (e.g., packets) received by a specific VNIC implemented / executed by the specific NVD is redirected to the shadow VNIC instead. The shadow VNIC can be implemented by a system of one or more devices different from the specific NVD. In this way, traffic is diverted from the NVD potentially under a DDoS attack to a different device or system, referred to as the DDoS scrubber system. In this way, the specific NVD under attack is protected.

[0059] After continuously monitoring a network resource that has entered the protected mode, it can be determined that the network resource can be converted from the protected mode to the normal operation mode. In the normal operation mode, the shadow VNICs created for the VNIC or NVD are deleted, and information about the original VNIC is published. Thus, the communication of network traffic (e.g., packets) received by the original VNIC is restored without any redirection.

[0060] The novel techniques disclosed in this disclosure provide several benefits. After detecting a potential DDoS attack, ONDMS enters a protected mode to protect specific network resources. The protected mode utilizes a highly scalable DDoS scrubber system that includes multiple machines (e.g., a team of host machines) configured to execute or implement one or more shadow VNICs to perform mitigation actions. After the potential DDoS attack no longer exists, ONDMS can switch back to the normal operation mode. The DDoS mitigation process is transparent to the customer and requires no operator intervention.

[0061] For another benefit, imagine a scenario without a separate shadow VNIC. The protected VNIC implemented by NVD would need to perform filtering to send only 1 Gbps to each customer's compute instance under the bandwidth of that VNIC. However, when the aggregated traffic going to the protected VNIC reaches 100 Gbps for the 1 Gbps receiving limit of that VNIC, the protected VNIC may become overloaded. This situation means that no other compute instances will be able to receive traffic because each compute instance that should receive 1 Gbps out of the 100 Gbps on the network link is now receiving nothing. Additionally, due to the ineffective filtering of the protected VNIC, packets of other customers sharing the same compute instance may be dropped and affected. Therefore, using shadow VNICs implemented by a DDoS scrubber system with more computing power, bandwidth, and dedicated mitigation functions (e.g., dropping, filtering, and throttling) can reduce the impact of noisy neighbors.

[0062] Finally, these novel techniques do not need to wait until network traffic problems have been analyzed and determined before taking action, but can proactively redirect network traffic to the SD-VNIC to perform additional filtering. For example, when a DDoS attack initially affects a part of the CSPI and causes a noisy neighbor problem, passive reaction and network analysis may require more work to identify the cause. The techniques described in this disclosure can take proactive actions by monitoring network traffic and performing DDoS attack mitigation to prevent further interference.

[0063] Figures 1 - 5 The associated description provided in the following "Example Virtual Networking Architecture" section describes networking concepts including network virtualization, substrate network, overlay network, VNIC, etc., and provides an example of an environment in which certain embodiments described in this disclosure can be implemented. Figures 6 - 9 Examples and embodiments related to the ONDMS architecture described in this disclosure are described. Figures 10 - 13 An example of an architecture for implementing a cloud infrastructure for providing one or more cloud services is depicted, where the infrastructure can incorporate the teachings described herein. Figure 14Depicts a block diagram of an example computer system or device according to at least one embodiment.

[0064] Example Virtual Networking Architecture

[0065] The term cloud service is generally used to refer to services that are available on demand (e.g., via a subscription model) to users or customers by a cloud service provider (CSP) using systems and infrastructure (cloud infrastructure) provided by the CSP. Generally, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Thus, a customer can utilize the cloud services provided by the CSP on its own without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide subscribing customers with simple, scalable access to applications and computing resources without the customer having to invest in the infrastructure for providing the services.

[0066] There are several cloud service providers that offer various types of cloud services. There are various different types or models of cloud services, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.

[0067] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service provided by a CSP, a lease or account is created for that customer. The customer can then access the one or more subscribed cloud resources associated with that account via this account.

[0068] As described above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, the CSP provides infrastructure (referred to as cloud service provider infrastructure or CSPI) that can be used by customers to build their own customizable networks and deploy customer resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0069] A CSPI may include interconnected high-performance computing resources (including various host machines, memory resources, and network resources) that form a physical network, which is also referred to as a substrate network or an underlying network. The resources in a CSPI may be spread across one or more data centers, which may be geographically spread across one or more geographical regions. Virtualization software may be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also referred to as a software-based network, a software-defined network, or a virtual network) on top of the physical network. The CSPI physical network provides the underlying foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or substrate network or underlying network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that runs on top of the physical substrate network. A given physical network may support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a Virtual Cloud Network (VCN). A virtual network is implemented using software virtualization techniques (e.g., a hypervisor, virtualization functions implemented by a Network Virtualization Device (NVD) (e.g., a smartNIC), a Top-of-Rack (TOR) switch, a Smart TOR that implements one or more functions performed by an NVD, and other mechanisms) to create a layer of network abstraction that can run on top of the physical network. A virtual network may take many forms, including peer-to-peer networks, IP networks, etc. A virtual network is typically a layer-3 IP network or a layer-2 VLAN. This method of virtual or overlay networking is often referred to as virtual or overlay layer-3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC 7348), Virtual Private Network (VPN) (e.g., MPLS layer-3 Virtual Private Network (RFC4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0070] For IaaS, the infrastructure provided by a CSP (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., the hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, because these services can be policy-driven, IaaS users can implement policy-driven load balancing to maintain application availability and performance. CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, managed, distributed environment. CSPI delivers high-performance computing resources and capabilities, as well as storage capacity, in a flexible virtual network that is securely accessible from a variety of networked locations, such as from the customer's on-premises network. When a customer subscribes to or signs up for an IaaS service provided by a CSP, the tenancy created for that customer is a secure and isolated partition within CSPI where the customer can create, organize, and manage their cloud resources.

[0071] Customers can use the compute, storage, and networking resources provided by CSPI to build their own virtual networks. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, customers can use the resources provided by CSPI to build one or more customizable and private virtual networks, called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, on the customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Thus, CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosted environment. Customers do not manage or control the underlying physical resources provided by CSPI, but do have control over the operating system, storage devices, and deployed applications; and may also have limited control over selected networking components (e.g., firewalls).

[0072] The CSP may provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In some embodiments, the console provides a web-based user interface that can be used to access and manage CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0073] CSPI can support single-tenant or multi-tenant architectures. In a single-tenant architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenant architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenant architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenant scenario, preventive measures are taken and protection measures are implemented within CSPI to ensure that each tenant's data is isolated and invisible to other tenants.

[0074] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to the physical network and communicates back and forth with the network to which the network endpoint is connected. Network endpoints in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer-2 address (e.g., MAC address), a fixed layer-3 address (e.g., IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as an overlay layer-2 address (e.g., overlay MAC address) and an overlay layer-3 address (e.g., overlay IP address). Network overlay enables flexibility by allowing network administrators to move around the overlay addresses associated with network endpoints using software management (e.g., via software that implements a control plane for the virtual network). Accordingly, different from a physical network, in a virtual network, an overlay address (e.g., overlay IP address) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communication between components in a virtual network involves both the virtual network and the underlying physical network. To facilitate such communication, components of CSPI are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa. These mappings are then used to facilitate communication. Customer traffic is encapsulated to facilitate routing in the virtual network.

[0075] Accordingly, a physical address (e.g., a physical IP address) is associated with a component in a physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in a substrate or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in an overlay network, such as an overlay address associated with a compute instance in a customer's Virtual Cloud Network (VCN). Two different customers or tenants (each with their own private VCN) can potentially use the same overlay IP address in their VCNs without knowing about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These addresses are independent of virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer can use a VIP to map or represent multiple servers, each with its own real IP address.

[0076] The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions of the world. The CSPI can include components in a physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, the CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a localized geographic area that contains one or more data centers. Regions are generally independent of each other and can be far apart, e.g., spanning countries or even continents. For example, a first region can be in Australia, another in Japan, and yet another in India, etc. CSPI resources are divided among the regions such that each region has its own independent subset of CSPI resources. Each region can provide a collection of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and associated infrastructure, etc.); storage resources (e.g., block volume storage, file storage, object storage, archival storage); networking resources (e.g., Virtual Cloud Network (VCN), load balancing resources, connection to an on-premises network), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region generally has multiple paths that connect it to other regions in the realm.

[0077] Generally, an application is deployed in the region where it is most frequently used (i.e., deployed on the infrastructure associated with that region) because using nearby resources is faster than using distant resources. An application can also be deployed in different regions for various reasons, such as redundancy to mitigate the risk of region-wide events (such as large weather systems or earthquakes), to meet different requirements such as legal jurisdictions, tax domains, and other commercial or social criteria.

[0078] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within the region. A region can consist of one or more Availability Domains. In such a distributed environment, CSPI resources are either region-specific (such as Virtual Cloud Networks (VCNs)) or Availability Domain-specific (such as compute instances).

[0079] The ADs within a region are isolated from each other, fault-tolerant, and configured such that it is highly unlikely that they will all fail simultaneously. This is achieved by the ADs not sharing critical infrastructure resources (such as networking, physical cables, cable paths, cable entry points, etc.), such that a failure at one AD within a region is unlikely to affect the availability of other ADs within the same region. The ADs within the same region can be connected to each other via a low-latency, high-bandwidth network, which makes it possible to provide highly available connectivity to other networks (e.g., the Internet, a customer's on-premises network, etc.) and to build replicated systems across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by an IaaS provider grows, more regions and ADs with additional capacity can be added. Traffic between Availability Domains is typically encrypted.

[0080] In some embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot. A customer's lease or account with a CSP exists within a single realm and can be spread across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a lease or account for that customer is created in the region (referred to as the "primary" region) specified by the customer within the realm. The customer can extend the customer's lease across one or more other regions within the realm. A customer cannot access regions that are not within the realm where the customer's lease is located.

[0081] IaaS providers can offer multiple realms, each realm meeting the needs of a specific set of customers or users. For example, a business realm can be provided for business customers. As another example, for customers within a specific country, a realm can be provided for that specific country. As yet another example, a government realm can be provided for the government, and so on. For example, a government realm can meet the needs of a specific government and can have a higher security level than a business realm. For example, Oracle Cloud Infrastructure (OCI) currently offers realms for commercial regions and two realms for government cloud regions (e.g., FedRAMP authorized and IL5 authorized).

[0082] In some embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow the distribution of compute instances such that these instances do not reside on the same physical hardware within a single AD. This is referred to as anti-affinity. A fault domain refers to a collection of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Thus, a hardware failure or a compute hardware maintenance event affecting one fault domain does not affect the instances in other fault domains. Depending on the embodiment, the number of fault domains for each AD can vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0083] When a customer subscribes to an IaaS service, resources from the CSPI are provisioned to the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. The customer network hosted by the CSPI in the cloud is referred to as a Virtual Cloud Network (VCN). The customer can use the CSPI resources allocated to the customer to set up one or more Virtual Cloud Networks (VCNs). A VCN is a virtual or software-defined private network. Customer resources deployed in the customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints ("public endpoints") over a public network (such as the Internet), with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs that do not belong to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0084] A CSP can use CSPI to provide various services. In some cases, the customers of CSPI themselves can act like service providers and use CSPI resources to provide services. The service provider can expose service endpoints characterized by identification information (e.g., IP address, DNS name, and port). The customer's resources (e.g., computing instances) can use a particular service by accessing the service endpoint exposed by the service for that particular service. These service endpoints are generally endpoints that can be publicly accessed by users via a public communication network such as the Internet using the public IP address associated with the endpoint. The publicly accessible network endpoints are sometimes also referred to as public endpoints.

[0085] In some embodiments, the service provider can expose the service via an endpoint for the service (sometimes referred to as a service endpoint). The customers of the service can then use this service endpoint to access the service. In some implementations, the service endpoints provided for a service can be accessed by multiple customers intending to consume the service. In other implementations, dedicated service endpoints can be provided for a customer such that only that customer can use the dedicated service endpoint to access the service.

[0086] In some embodiments, when a VCN is created, the VCN is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a series of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, routing tables, and gateways. The VCN resides within a single region but can span one or more or all of the availability domains within that region. A gateway is a virtual interface configured for the VCN and enables traffic to be transmitted to and from the VCN to one or more endpoints external to the VCN. One or more different types of gateways can be configured for the VCN to enable communication to and from different types of endpoints.

[0087] A VCN can be subdivided into one or more sub-networks, such as one or more subnets. Thus, a subnet is a configured unit or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of the address space within the VCN's address space.

[0088] Each compute instance is associated with a virtual network interface card (VNIC), which enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally speaking, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet, has one or more associated IP addresses, and associated security rules or policies. A VNIC is analogous to a layer-2 port on a switch. A VNIC is attached to a compute instance and a subnet within a VCN. The VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in different subnets within the VCN, or with endpoints external to the VCN. Thus, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with the compute instance. For a subnet that includes a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, with each VNIC attached to a compute instance within the set of compute instances.

[0089] A private overlay IP address is assigned to each compute instance via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of the address space within the VCN's address space. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses allocated for that subnet.

[0090] In some embodiments, in addition to the private overlay IP address, a compute instance can optionally be assigned additional overlay IP addresses, such as, for example, one or more public IP addresses if in a public subnet. These multiple addresses are assigned either on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created during instance startup and is associated with the overlay private IP address assigned to the instance - this primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. A secondary VNIC can be in a subnet in the same VCN as the primary VNIC or in a different subnet in either the same VCN or a different VCN.

[0091] If a compute instance is in a public subnet, it can optionally be assigned a public IP address. When creating a subnet, the subnet can be designated as a public subnet or a private subnet. A private subnet means that resources (e.g., compute instances) in the subnet and the associated VNICs cannot have public overlay IP addresses. A public subnet means that resources and the associated VNICs in the subnet can have public IP addresses. A customer can specify that the subnet exists in a single availability domain or across multiple availability domains in a region or realm.

[0092] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as the VCN VR or simply the VR) enables communication between the subnets of the VCN. For a subnet within a VCN, the VR represents the logical gateway for that subnet, which enables the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN as well as with endpoints outside the VCN. The VCN VR is a logical entity that is configured to route traffic between the VNICs in the VCN and the virtual gateway associated with the VCN ("gateway"). Below regarding Figure 1Describe the gateway further. VCN VR is a layer-3 / IP layer concept. In one embodiment, there is a VCN VR for a VCN, where the VCN VR potentially has an unrestricted number of ports addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address from the overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN having two subnets with associated address ranges of 10.0 / 16 and 10.1 / 16 respectively. For the first subnet within the VCN with an address range of 10.0 / 16, an address from this range is reserved for the port of the VCN VR for that subnet. In some cases, the first IP address from the range can be reserved for the VCN VR. For example, for a subnet with an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 can be reserved for the port of the VCN VR for that subnet. For the second subnet in the same VCN with an address range of 10.1 / 16, the VCN VR can have a port for that second subnet with an IP address of 10.1.0.1. The VCN VR has a different IP address for each subnet in the VCN.

[0093] In some other embodiments, each subnet within a VCN can have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with the VR . For example, the reserved or default IP address can be the first IP address from the IP address range associated with the subnet. VNICs within the subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such an embodiment, the VR is the entry / exit point for the subnet. The VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with the gateways associated with the VCN. The VR functionality for a subnet runs on or is performed by one or more NVDs that perform the VNIC functionality for the VNICs within the subnet.

[0094] A routing table, security rules, and DHCP options can be configured for the VCN. The routing table is the virtual routing table for the VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. The routing table for the VCN can be customized to control how packets are forwarded / routed into and out of the VCN. DHCP options refer to the configuration information that is automatically provided to an instance when the instance is launched.

[0095] The security rules configured for a VCN represent an overlay firewall rule for the VCN. The security rules can include ingress and egress rules and specify the types of traffic (e.g., based on protocol and port) that are allowed to flow in and out of instances within the VCN. The customer can choose whether a given rule is stateful or stateless. For example, the customer can allow incoming SSH traffic from anywhere to a set of instances by establishing a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. The security rules can be implemented using network security groups or security lists. A network security group consists of a collection of security rules that apply only to the resources within that group. On the other hand, a security list includes rules that apply to all resources in any subnet that uses that security list. A default security list with default security rules can be provided for the VCN. The DHCP options configured for the VCN provide configuration information that is automatically provided to the instances within the VCN when they are launched.

[0096] In some embodiments, the configuration information for a VCN is determined and stored by the VCN control plane. For example, the configuration information for a VCN can include information about the address range associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, compute instances in the VCN and associated VNICs, NVDs (e.g., VNICs, VRs, gateways) that perform various virtualized network functions associated with the VCN, status information for the VCN, and other VCN-related information. In some embodiments, the VCN distribution service publishes the configuration information stored by the VCN control plane or portions thereof to the NVDs. The distributed information can be used to update the information (e.g., forwarding tables, routing tables, etc.) stored by the NVDs and used for forwarding packets to and from compute instances in the VCN.

[0097] In some embodiments, the creation of the VCN and subnets is handled by the VCN control plane (CP) and the launch of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instances and then invoking the VCN control plane to create VNICs and attach them to the compute instances. The VCN CP also sends the VCN data map to the VCN data plane that is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of the VCN control plane are also depicted in Figure 11 、 Figure 12 、 Figure 13 and Figure 14 (see reference numerals 1116, 1216, 1316, and 1416) and are described below.

[0098] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on the customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside CSPI.

[0099] Various different architectures for implementing cloud-based services using CSPI are depicted in Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 11 、 Figure 12 、 Figure 13 and Figure 15 and are described below. Figure 1 is a high-level diagram of a distributed environment 100 of an overlay or customer VCN hosted by CSPI according to certain embodiments. Figure 1 The distributed environment depicted in Figure 1 includes multiple components in an overlay network. The distributed environment 100 depicted in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, Figure 1 the distributed environment depicted in

[0100] As shown in the example depicted in Figure 1 the distributed environment 100 includes CSPI 101 that provides services and resources that customers can subscribe to and use to build their virtual cloud networks (VCNs). In certain embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. Figure 1 An example region "Region US" 102 is shown in

[0101] In Figure 1 the depicted embodiment, the customer VCN 104 includes two subnets, namely, "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. In Figure 1Among them, the covered IP address range of Subnet-1 is 10.0 / 16, and the address range of Subnet-2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway for the VCN. This VCN virtual router enables communication between the subnets of VCN 104 and other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between the VNICs in VCN 104 and the gateways associated with VCN 104. The VCN VR 105 provides ports for each subnet of VCN 104. For example, VR 105 can provide a port with the IP address 10.0.0.1 for Subnet-1 and a port with the IP address 10.1.0.1 for Subnet-2.

[0102] Multiple computing instances can be deployed on each subnet, where the computing instances can be virtual machine instances and / or bare metal instances. The computing instances in the subnet can be hosted by one or more host machines within CSPI 101. The computing instances participate in the subnet via the VNIC associated with the computing instance. For example, as Figure 1 shown in, the computing instance C1 becomes part of Subnet-1 via the VNIC associated with the computing instance. Similarly, the computing instance C2 becomes part of Subnet-1 via the VNIC associated with C2. In a similar manner, multiple computing instances (which can be virtual machine instances or bare metal instances) can be part of Subnet-1. Via its associated VNIC, each computing instance is assigned a private covered IP address and a MAC address. For example, in Figure 1 it is shown that the covered IP address of computing instance C1 is 10.0.0.2 and the MAC address is M1, while the private covered IP address of computing instance C2 is 10.0.0.3 and the MAC address is M2. Each computing instance in Subnet-1 (including computing instances C1 and C2) has a default route to the VCN VR 105 using the IP address 10.0.0.1, which is the IP address of the port of VCN VR 105 for Subnet-1.

[0103] Multiple computing instances can be deployed on Subnet-2, including virtual machine instances and / or bare metal instances. For example, as Figure 1 shown in, the computing instances D1 and D2 become part of Subnet-2 via the VNICs associated with the respective computing instances. In Figure 1In the depicted embodiment, the overlay IP address of compute instance D1 is 10.1.0.2 and the MAC address is MM1, while the private overlay IP address of compute instance D2 is 10.1.0.3 and the MAC address is MM2. Each compute instance in Subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR105 for Subnet-2.

[0104] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets in the VCN.

[0105] A particular compute instance deployed on VCN 104 may communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 101 may include: endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet-1); endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in Subnet-1 and a compute instance in Subnet-2); endpoints in different VCNs in the same region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet-1 and an endpoint in the service network 110 in the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in subnets hosted by CSPI 101 may also communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in the customer's on-premises network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0106] Use the VNICs associated with the source and destination compute instances to facilitate communication between compute instances on the same subnet. For example, a compute instance C1 in subnet-1 may want to send a packet to a compute instance C2 in subnet-1. For a packet originating from a source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the destination information of the packet from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop with the aim of facilitating the delivery of the packet to the intended destination of the packet. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then receives and processes the packet and forwards the packet to the destination compute instance.

[0107] For packets to be delivered from a compute instance in a subnet to an endpoint in a different subnet within the same VCN, communication is facilitated through the VNICs associated with the source and destination compute instances and the VCN VR. For example, if Figure 1 compute instance C1 in subnet-1 in wants to send a packet to compute instance D1 in subnet-2, then the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route of the VCN VR or port 10.0.0.1. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1 and the VNIC forwards the packet to compute instance D1.

[0108] For packets to be transferred from a compute instance in the VCN 104 to an endpoint outside the VCN 104, the communication is facilitated by the VNIC associated with the source compute instance, the VCN VR 105, and the gateway associated with the VCN 104. One or more types of gateways can be associated with the VCN 104. A gateway is an interface between the VCN and another endpoint, where that other endpoint is outside the VCN. A gateway is a layer-3 / IP layer concept and enables the VCN to communicate with endpoints outside the VCN. Thus, the gateway facilitates the flow of traffic between the VCN and other VCNs or networks. Various different types of gateways can be configured for the VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication can be through a public network (e.g., the Internet) or through a private network. Various communication protocols can be used for these communications.

[0109] For example, compute instance C1 may want to communicate with an endpoint outside the VCN 104. The packet can first be processed by the VNIC associated with the source compute instance C1. The VNIC processing determines that the destination of the packet is outside subnet-1 of C1. The VNIC associated with C1 can forward the packet to the VCN VR 105 for the VCN 104. Then the VCN VR 105 processes the packet and, as part of the processing, determines a specific gateway associated with the VCN 104 as the next hop for the packet based on the destination of the packet. Then, the VCN VR 105 can forward the packet to the identified specific gateway. For example, if the destination is an endpoint within the customer's on-premises network, then the packet can be forwarded by the VCN VR 105 to the Dynamic Routing Gateway (DRG) gateway 122 configured for the VCN 104. Then the packet can be forwarded from the gateway to the next hop to facilitate the delivery of the packet to its final intended destination.

[0110] Various different types of gateways can be configured for the VCN. Examples of gateways that can be configured for the VCN are depicted in Figure 1 and described below. Examples of gateways associated with the VCN are also depicted in Figure 11 , Figure 12 , Figure 13 and Figure 14 and described below (e.g., the gateways referenced by reference numerals 1134, 1136, 1138, 1234, 1236, 1238, 1334, 1336, 1338, 1434, 1436, and 1438). As Figure 1As shown in the embodiments depicted, a Dynamic Routing Gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provide a path for private network traffic communication between the customer VCN 104 and another endpoint, where the other endpoint can be the customer's on-premises network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer on-premises network 116 can be a customer network or customer data center built using the customer's resources. Access to the customer on-premises network 116 is generally very restricted. For a customer having both a customer on-premises network 116 and one or more VCNs 104 deployed or hosted by the CSPI 101 in the cloud, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This enables the customer to build an extended hybrid environment encompassing the customer's VCN 104 hosted by the CSPI 101 and their on-premises network 116. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is established, where one endpoint of the channel is in the customer on-premises network 116 and the other endpoint is located in the CSPI 101 and connected to the customer VCN 104. The communication channel 124 can be through a public communication network (such as the Internet) or a private communication network. Various different communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology that uses a private network instead of a public network, etc. The device or equipment forming one endpoint of the communication channel 124 in the customer on-premises network 116 is referred to as customer premise equipment (CPE), such as Figure 1 the CPE 126 depicted in

[0111] In some embodiments, a Remote Peering Connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, the customer VCN 104 can be connected to a VCN 108 in another region using the DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101 (such as the Microsoft Azure cloud, the Amazon AWS cloud, etc.).

[0112] As Figure 1As shown, an Internet Gateway (IGW) 120 can be configured for user VCN 104, which enables computing instances on VCN 104 to communicate with public endpoints 114 accessible via a public network such as the Internet. IGW 120 is a gateway that connects the VCN to a public network such as the Internet. IGW 120 enables public subnets within the VCN (such as VCN 104), where resources in the public subnets have public overlay IP addresses, to directly access public endpoints 112 on the public network 114 (such as the Internet). Using IGW 120, connections can be initiated from subnets within VCN 104 or from the Internet.

[0113] A Network Address Translation (NAT) Gateway 128 can be configured for the customer's VCN 104 and enables cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, and it does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables private access from a private subnet within the VCN (such as Private Subnet-1 in VCN104) to public endpoints on the Internet. In a NAT gateway, connections to the public Internet can only be initiated from the private subnet and not from the Internet to the private subnet.

[0114] In some embodiments, a Service Gateway (SGW) 126 can be configured for customer VCN 104 and provides a path for private network traffic between VCN104 and service endpoints supported in the service network 110. In some embodiments, the service network 110 can be provided by a CSP and can provide various services. An example of such a service network is Oracle's service network, which provides various services that can be used by customers. For example, a computing instance (such as a database system) in a private subnet of customer VCN 104 can back up data to a service endpoint (such as object storage) without the need for a public IP address or access to the Internet. In some embodiments, a VCN can have only one SGW, and the connection can only be initiated from a subnet within the VCN and not from the service network 110. If a VCN is peered with another VCN, resources in that other VCN generally cannot access the SGW. Resources in an on-premises network connected to the VCN using FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0115] In some embodiments, the SGW 126 uses the concept of a service Classless Inter-Domain Routing (CIDR) label, which is a string representing all the regional public IP address ranges for a service or group of services of interest. Customers use the service CIDR label when they configure the SGW and associated routing rules to control traffic to the service. Customers can optionally use the service CIDR label when configuring security rules, without the need to adjust the security rules in the future if the public IP address of the service changes.

[0116] The Local Peering Gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 and enables the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without traffic crossing a public network (such as the Internet) or traffic being routed through the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each peer it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0117] Service providers (such as providers of services in the service network 110) can provide access to services using different access models. According to the public access model, the service can be exposed as a public endpoint that can be publicly accessed by compute instances in the customer VCN via a public network (such as the Internet), and / or can be privately accessed via the SGW 126. According to a specific private access model, the service can be accessed as a private IP endpoint in a private subnet within the customer's VCN. This is called Private Endpoint (PE) access and enables service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE appears as a VNIC (referred to as a PE-VNIC, having one or more private IPs) in a subnet chosen by the customer within the customer's VCN. Thus, the PE provides a way to present the service as a VNIC in a private customer VCN subnet. Since the endpoint is exposed as a VNIC, all the features associated with the VNIC (such as routing rules, security lists, etc.) can now be used for the PE VNIC.

[0118] Service providers can register their services to enable access via the PE. The provider can associate policies with the service, which limit the visibility of the service to the customer's tenancy. The provider can register multiple services under a single Virtual IP Address (VIP), especially for multi-tenant services. There can be multiple such private endpoints (in multiple VCNs) representing the same service.

[0119] Compute instances in the private subnet can then access the service using the private IP address of the PE VNIC or the service DNS name. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in the service network 110), and the PAGW acts as an ingress / egress point for all traffic from / to the private endpoints of the customer subnets. The PAGW 130 enables the provider to scale the number of PE connections without utilizing its internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can represent the service as private endpoints in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC is not attached to the customer's instance but appears to be attached to the service that the customer wishes to interact with. Traffic sent to the private endpoint is routed to the service via the PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0120] By allowing traffic to flow through the FastConnect / IPsec link and the private endpoints in the customer VCN, the PE concept can also be used to extend private access to the service to the customer's on-premises networks and data centers. By allowing traffic to flow between the LPG 132 and the PE in the customer's VCN, private access to the service can also be extended to the customer's peered VCNs.

[0121] The customer can control routing in the VCN at the subnet level, so the customer can specify which subnets in the customer's VCN (such as VCN104) use each gateway. The routing table of the VCN is used to decide whether to allow traffic to leave the VCN through a specific gateway. For example, in a specific instance, the routing table for the public subnet within the customer VCN 104 can send non-local traffic through the IGW 120. The routing table for the private subnet within the same customer VCN 104 can send traffic destined for the CSP service through the SGW 126. All remaining traffic can be sent via the NAT gateway 128. The routing table only controls traffic leaving the VCN.

[0122] The security list associated with the VCN is used to control the traffic entering the VCN via the gateway through the inbound connection. All resources in a subnet use the same route table and security list. The security list can be used to control the specific types of traffic allowed to and from the instances in the subnet of the VCN. The security list rules can include ingress (inbound) and egress (outbound) rules. For example, the ingress rule can specify the allowed source address range, while the egress rule can specify the allowed destination address range. The security rules can specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the operating system of the instance can enforce its own firewall rules that align with the security list rules. The rules can be stateful (e.g., tracking connections and automatically allowing responses without an explicit security list rule for the response traffic) or stateless.

[0123] Access from the customer VCN (i.e., through resources or compute instances deployed on VCN 104) can be classified as public access, private access, or dedicated access. Public access refers to an access model that uses a public IP address or NAT access to a public endpoint. Private access enables customer workloads with private IP addresses in VCN 104 (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access the public service endpoints of services using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoints of the services residing outside the customer's private network.

[0124] Additionally, CSPI can use technologies such as FastConnect public peering to provide dedicated public access, where customer on-premises instances can use a FastConnect connection to access one or more services in the customer VCN without traversing a public network such as the Internet. CSPI can also use FastConnect private peering to provide dedicated private access, where customer on-premises instances with private IP addresses can use a FastConnect connection to access the customer's VCN workloads. FastConnect is a network connectivity alternative to using the public Internet to connect the customer's on-premises network to CSPI and its services. Compared to Internet-based connections, FastConnect provides a simple, flexible, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent networking experience.

[0125] Figure 1The above and the accompanying description describe various virtualized components in an example virtual network. As described above, the virtual network is built on an underlying physical or substrate network. Figure 2 FIG. 2 depicts a simplified architecture diagram of the physical components in a physical network that provides the underlying for a virtual network within a CSPI 200 according to certain embodiments. As shown, the CSPI 200 provides a distributed environment that includes components and resources (e.g., computing, memory, and networking resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed to by the customer, a subset of the resources (e.g., computing, memory, and networking resources) of the CSPI 200 are provisioned to the customer. The customer can then use the physical computing, memory, and networking resources provided by the CSPI 200 to build their own cloud-based (i.e., CSPI-hosted) customizable and private virtual network. As previously indicated, these customer networks are referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as computing instances, on these customer VCNs. The computing instances can be in the form of virtual machines, bare metal instances, etc. The CSPI 200 provides a collection of infrastructure and complementary cloud services that enable the customer to build and run a wide range of applications and services in a highly available hosted environment.

[0126] In Figure 2 the example embodiment depicted in FIG. 3, the physical components of the CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches in the physical network 218. The physical host machines or servers can host and execute various computing instances that participate in one or more subnets of the VCN. The computing instances can include virtual machine instances and bare metal instances. For example, Figure 1 the various computing instances depicted in FIG. 4 can be hosted by Figure 2 the physical host machines depicted in FIG. 5. The virtual machine computing instances in the VCN can be executed by one host machine or multiple different host machines. The physical host machines can also host virtual host machines, container-based hosts, functions, etc. Figure 1 The VNICs and VCN VRs depicted in FIG. 6 can be executed by Figure 2 the NVDs depicted in FIG. 7. Figure 1 The gateways depicted in FIG. 8 can be executed by the host machines and / or by Figure 2 the NVDs depicted in FIG. 9.

[0127] A host machine or server can execute a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more computing instances can be created, executed, and managed on the host machine by the hypervisor on that host machine. The hypervisor on the host machine enables the physical computing resources of the host machine (e.g., computing, memory, and networking resources) to be shared among the various computing instances executed by the host machine.

[0128] For example, as Figure 2 depicted in, host machines 202 and 208 execute hypervisors 260 and 266 respectively. These hypervisors can be implemented using software, firmware, hardware, or a combination thereof. Generally, a hypervisor is a process or software layer that sits on top of the operating system (OS) of the host machine, which in turn executes on the hardware processor of the host machine. The hypervisor provides a virtualized environment by enabling the physical computing resources of the host machine (e.g., processing resources such as processors / cores, memory resources, networking resources) to be shared among the various virtual machine computing instances executed by the host machine. For example, in Figure 2 , hypervisor 260 can sit on top of the OS of host machine 202 and enable the computing resources of host machine 202 (e.g., processing, memory, and networking resources) to be shared among the computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own operating system (referred to as a guest operating system), which can be the same as or different from the OS of the host machine. The operating system of a virtual machine executed by a host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor enables multiple operating systems to be executed simultaneously while sharing the same computing resources of the host machine. Figure 2 The host machines depicted in can have the same or different types of hypervisors.

[0129] A computing instance can be a virtual machine instance or a bare-metal instance. In Figure 2 , computing instances 268 on host machine 202 and 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare-metal instance provided to a customer.

[0130] In some cases, an entire host machine can be provisioned to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine all belong to the same customer. In other cases, the host machine can be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs of different customers. In some embodiments, the bare metal compute instances are hosted by bare metal servers without a hypervisor. When provisioning a bare metal compute instance, a single customer or tenant maintains control over the physical CPUs, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0131] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates the transfer of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In some embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 the embodiment depicted, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0132] For a compute instance hosted by a host machine, the NVD connected to the host machine also executes the VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment depicted in Figure ..... NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 2i2 can also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0133] The host machine can include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NICs on the host machine can provide one or more ports (or interfaces) that enable the host machine to communicatively connect to another device. For example, the host machine can connect to an NVD using one or more ports (or interfaces) provided on the host machine and on the NVD. The host machine can also connect to other devices (such as another host machine).

[0134] For example, in ​ , host machine 202 connects to NVD 210 using link 220, which extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 connects to NVD 212 using link 224, which extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 connects to NVD 212 using link 226, which extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0135] The NVDs are in turn connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also referred to as a switch fabric). In some embodiments, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in ​ , NVDs 210 and 212 connect to TOR switches 214 and 216 using links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0136] The physical network 218 provides a communication fabric that enables the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In some implementations, the physical network 218 is a multi-tier Clos network of switches, where TOR switches 214 and 216 represent the leaf-level nodes of the multi-tier and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and general “n”-tier networks. Examples of Clos networks are depicted in ​ and described below.

[0137] There can be various different connection configurations between the host machine and the NVD, such as one-to-one configuration, many-to-one configuration, one-to-many configuration, etc. In a one-to-one configuration implementation, each host machine is connected to its own separate NVD. For example, in ​ host machine 202 is connected to NVD 210 via the NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in ​ host machines 206 and 208 are respectively connected to the same NVD 212 via NICs 244 and 250.

[0138] In a one-to-many configuration, one host machine is connected to multiple NVDs. ​ An example within the CSPI 300 is shown where a host machine is connected to multiple NVDs. As ​ shown, host machine 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and is connected to a second NVD 312 via port 308 and link 322. Ports 306 and 308 can be Ethernet ports, and the links 320 and 322 between host machine 302 and NVDs 310 and 312 can be Ethernet links. NVD 310 is further connected to a first TOR switch 314 and NVD 312 is connected to a second TOR switch 316. The links between NVDs 310, 312 and TOR switches 314, 316 can be Ethernet links. TOR switches 314 and 316 represent level 0 switching devices in a multi-level physical network 318.

[0139] ​ The arrangement depicted in

[0140] In ​In the configuration depicted, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs enabling connectivity of the host machine to multiple NVDs.

[0141] Return reference ​ , an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device having one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline, etc.), memory including a cache, and ports. The various virtualization functions can be performed by software / firmware executed by one or more of the NVD's processing units.

[0142] An NVD can be implemented in a variety of different forms. For example, in some embodiments, the NVD is implemented as an interface card called a smartNIC or a smart NIC with an on-board embedded processor. A smartNIC is a device separate from the NIC on the host machine. In ​ , the NVDs 210 and 212 can be implemented as smartNICs respectively connected to the host machine 202 and the host machines 206 and 208.

[0143] However, a smartNIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD can be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of the CSPI 200. For example, the NVD can be implemented in a host machine, where the functions performed by the NVD are performed by the host machine. As another example, the NVD can be part of a TOR switch, or the TOR switch can be configured to perform the functions performed by the NVD, which enables the TOR switch to perform various complex packet conversions for a public cloud. A TOR that performs the functions of an NVD is sometimes referred to as a smart TOR. In other implementations where virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD can be implemented inside the hypervisor of the host machine. In some other embodiments, some of the functions of the NVD can be offloaded to a centralized service running on a cluster of host machines.

[0144] In certain embodiments, such as when implemented as in ​When referring to the smartNIC shown in , the NVD can include multiple physical ports that enable the NVD to connect to one or more host machines and to one or more TOR switches. The ports on the NVD can be classified as host-facing ports (also referred to as "south ports") or network-facing or TOR-facing ports (also referred to as "north ports"). The host-facing ports of the NVD are the ports used to connect the NVD to host machines. ​ Examples of host-facing ports in include port 236 on NVD 210 and ports 248 and 254 on NVD 212. The network-facing ports of the NVD are the ports used to connect the NVD to TOR switches. ​ Examples of network-facing ports in include port 256 on NVD210 and port 258 on NVD 212. As ​ shown in , NVD 210 uses link 228 extending from port 256 of NVD 210 to TOR switch 214 to connect to TOR switch 214. Similarly, NVD 212 uses link 230 extending from port 258 of NVD 212 to TOR switch 216 to connect to TOR switch 216.

[0145] The NVD receives packets and frames (e.g., packets and frames generated by compute instances hosted by the host machine) from the host machine via the host-facing ports, and after performing necessary packet processing, can forward the packets and frames to the TOR switch via the network-facing ports of the NVD. The NVD can receive packets and frames from the TOR switch via the network-facing ports of the NVD, and after performing necessary packet processing, can forward the packets and frames to the host machine via the host-facing ports of the NVD.

[0146] In some embodiments, there can be multiple ports and associated links between the NVD and the TOR switch. These ports and links can be aggregated to form a link aggregation group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG can operate at the same speed in full-duplex mode. LAG helps increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links within the LAG fails, then traffic will be dynamically and transparently re-assigned to one of the other physical links within the LAG. The aggregated physical links deliver higher bandwidth than each individual link. The multiple ports associated with the LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be between the NVD and the TOR switch, between the host machine and the NVD, etc.

[0147] The NVD implements or executes network virtualization functions. These functions are executed by the software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies, such as VCN security list (firewall) functionality; functions for facilitating the routing and forwarding of packets to and from compute instances in a VCN; and so on. In certain embodiments, after receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining how to forward or route the packet. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with an overlay network, such as executing a VNIC associated with a compute instance in a VCN, executing a virtual router (VR) associated with a VCN, encapsulating and decapsulating packets to facilitate forwarding or routing in a virtual network, executing certain gateways (e.g., local peer gateways), implementing security lists, network security groups, network address translation (NAT) functionality (e.g., translating public IPs to private IPs on a per-host basis), throttling functions, and other functions.

[0148] In certain embodiments, the packet processing data path in the NVD may include multiple packet pipelines, each packet pipeline consisting of a series of packet transformation stages. In certain implementations, after receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until the packet is discarded or sent out through an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., validating headers, enforcing throttling, inserting new layer-2 headers, enforcing L4 firewalls, VCN encapsulation / decapsulation, etc.) so that new pipelines can be constructed by combining existing stages, and new functionality can be added by creating new stages and inserting these stages into existing pipelines.

[0149] The NVD may execute both control plane and data plane functions corresponding to the control plane and data plane of a VCN. Examples of the VCN control plane are also depicted in ​ 、 ​ 、 ​ and ​ (see reference numerals 1116, 1216, 1316, and 1416) and described below. Examples of the VCN data plane are in ​ 、 ​ 、 ​ and ​Depicted (see reference numerals 1118, 1218, 1318, and 1418) and described below. Control plane functions include functions for configuring the network on how control data is forwarded (e.g., setting up routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN control plane is provided that centrally computes all overlay-to-substrate mappings and publishes these mappings to the NVD and virtual network edge devices (such as various gateways, such as DRG, SGW, IGW, etc.). The same mechanism can also be used to publish firewall rules. In some embodiments, the NVD only obtains the mappings relevant to that NVD. Data plane functions include functions for actually routing / forwarding packets based on the configuration established using the control plane. The VCN data plane is implemented by encapsulating the customer's network packets before they traverse the substrate network. The encapsulation / de-encapsulation functionality is implemented on the NVD. In some embodiments, the NVD is configured to intercept all network packets going in and out of the host machine and perform network virtualization functions.

[0150] As indicated above, the NVD performs various virtualization functions including VNIC and VCN VR. The NVD can perform the VNIC associated with a computing instance hosted by one or more host machines connected to the VNIC. For example, as ​ depicted, the NVD 210 performs the functionality of the VNIC 276 associated with the computing instance 268 hosted by the host machine 202 connected to the NVD 210. As another example, the NVD 212 performs the VNIC 280 associated with the bare-metal computing instance 272 hosted by the host machine 206 and performs the VNIC 284 associated with the computing instance 274 hosted by the host machine 208. The host machine can host computing instances belonging to different VCNs (these VCNs belong to different customers), and the NVD connected to the host machine can perform the VNIC corresponding to the computing instance (i.e., perform the functionality associated with the VNIC).

[0151] The NVD also performs the VCN virtual router corresponding to the VCN of the computing instance. For example, in ​ the depicted embodiment, the NVD 210 performs the VCN VR 277 corresponding to the VCN to which the computing instance 268 belongs. The NVD 212 performs one or more VCN VRs 283 corresponding to one or more VCNs to which the computing instances hosted by the host machines 206 and 208 belong. In some embodiments, the VCN VR corresponding to the VCN is performed by all NVDs connected to the host machine hosting at least one computing instance belonging to that VCN. If the host machine hosts computing instances belonging to different VCNs, then the NVD connected to that host machine can perform the VCN VR corresponding to that different VCN.

[0152] In addition to VNICs and VCN VRs, the NVD can execute various software (e.g., daemons) and includes components, where one or more of the hardware components facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as the ​ "packet processing components" shown in. For example, NVD 210 includes packet processing component 286 and NVD 212 includes packet processing component 288. For example, the packet processing components for the NVD can include a packet processor that is configured to interact with the ports and hardware interfaces of the NVD to monitor all packets received by and passed using the NVD and to store network information. The network information can include, for example, network flow information that identifies different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In some embodiments, the network flow information can be stored on a per-VNIC basis. The packet processor can perform per-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, the packet processing component can include a replication agent that is configured to copy information stored by the NVD to one or more different replication target repositories. As yet another example, the packet processing component can include a logging agent that is configured to perform the logging function of the NVD. The packet processing component can also include software for monitoring the performance and health of the NVD and also potentially the state and health of other components connected to the NVD.

[0153] ​ Shows the components of an example virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VR for the VCN, and a set of gateways configured for the VCN. ​ The overlay components depicted in can be executed or hosted by ​ one or more of the physical components depicted in. For example, a compute instance in a VCN can be executed or hosted by ​ one or more of the host machines depicted in. For a compute instance hosted by a host machine, the VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., VNIC functionality is provided by the NVD connected to that host machine). The VCN VR functionality for a VCN is executed by all NVDs connected to the host machine that hosts or executes a compute instance that is part of that VCN. Gateways associated with a VCN can be executed by one or more different types of NVDs. For example, some gateways can be executed by a smartNIC, while other gateways can be executed by one or more host machines or other implementations of the NVD.

[0154] As described above, the compute instances in the customer VCN can communicate with a variety of different endpoints, where the endpoints can be within the same subnet as the source compute instance, in a different subnet than the source compute instance but within the same VCN, or the endpoints are outside the VCN of the source compute instance. The VNIC associated with the compute instance, the VCN VR, and the gateways associated with the VCN are used to facilitate these communications.

[0155] For communications between two compute instances on the same subnet in a VCN, the VNICs associated with the source and destination compute instances are used to facilitate the communications. The source and destination compute instances can be hosted by the same host machine or different host machines. Packets originating from the source compute instance can be forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. At the NVD, a packet processing pipeline is used to process the packets, which can include executing the VNIC associated with the source compute instance. Since the destination endpoint of the packet is within the same subnet, executing the VNIC associated with the source compute instance causes the packet to be forwarded to the NVD that executes the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can be executed on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding table stored by the NVD to determine the next hop of the packet.

[0156] For packets to be delivered from a compute instance in a subnet to an endpoint in a different subnet within the same VCN, the packets originating from the source compute instance are passed from the host machine hosting the source compute instance to the NVD connected to that host machine. At the NVD, the packet processing pipeline is used to process the packets, which can include executing one or more VNICs and the VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes the functionality corresponding to the VNIC associated with the source compute instance (also referred to as executing the VNIC). The functionality executed by the VNIC can include looking at the VLAN tag on the packet. Since the destination of the packet is outside the subnet, the NVD then calls and executes the VCN VR functionality next. Then, the VCN VR routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source and destination compute instances can be executed on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0157] If the destination of the packet is outside the VCN of the source compute instance, then the packet originating from the source compute instance is passed from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes the VCN VR functionality, which may cause the packet to be forwarded to an NVD that executes the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, then the packet can be forwarded by the VCN VR to an NVD that executes the DRG gateway configured for the VCN. The VCN VR can be executed on the same NVD that executes the VNIC associated with the source compute instance, or by a different NVD. The gateway can be executed by an NVD, which can be a smartNIC, a host machine, or other NVD implementations. The packet is then processed by the gateway and forwarded to the next hop, which facilitates delivery of the packet to the destination endpoint expected by the packet. For example, in ​ the embodiment depicted in

[0158] compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by the CSPI 200 and endpoints external to the CSPI 200. Endpoints hosted by the CSPI 200 can include instances within the same VCN or other VCNs, which can be the customer's VCNs or VCNs that do not belong to the customer. Communication between endpoints hosted by the CSPI 200 can be performed over the physical network 218. Compute instances can also communicate with endpoints that are not hosted by the CSPI 200 or are external to the CSPI 200. Examples of such endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible via a public network such as the Internet. Communication with endpoints external to the CSPI 200 can be performed over a public network (e.g., the Internet) ( ​ not shown in ​ or a private network (

[0159] ​The architecture of the CSPI 200 depicted is merely exemplary and is not intended to be limiting. In alternative embodiments, variations, alternatives, and modifications are possible. For example, in some implementations, the CSPI 200 may have more or fewer systems or components than the ​ systems or components shown in, two or more systems may be combined, or it may have a different system configuration or arrangement. ​ The systems, subsystems, and other components depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device).

[0160] ​ Depicts the connectivity between a host machine and an NVD according to certain embodiments for providing I / O virtualization to support multi-tenancy. As ​ depicted in, the host machine 402 executes a hypervisor 404 that provides a virtualized environment. The host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to the NVD 412 via a link 414. Each computing instance is attached to a VNIC executed by the NVD 412. In ​ the embodiment of, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0161] As ​ shown in, the NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and is configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416, and VM2 408 is attached to logical NIC B 418. Although the host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, the virtual machines of each tenant believe they have their own host machine and NIC.

[0162] In some embodiments, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a different VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, the hypervisor attaches the tag assigned to tenant #1 to the packet, and then the packet is transmitted from the host machine 402 to the NVD 412 over link 414. In a similar manner, when a packet is transmitted from VM2 408, the hypervisor attaches the tag assigned to tenant #2 to the packet, and then the packet is transmitted from the host machine 402 to the NVD 412 over link 414. Accordingly, the packet 424 transmitted from the host machine 402 to the NVD 412 has an associated tag 426 that identifies the particular tenant and the associated VM. At the NVD, for the packet 424 received from the host machine 402, the tag 426 associated with the packet is used to determine whether the packet is to be processed by VNIC-VM1 420 or by VNIC-VM2 422. The packet is then processed by the corresponding VNIC. ​ The configuration depicted in ​ enables the compute instances of each tenant to believe that they have their own host machine and NIC. ​ The setup depicted in ​ provides I / O virtualization to support multi-tenancy.

[0163] ​ FIG. depicts a simplified block diagram of a physical network 500 in accordance with some embodiments. ​ The embodiment depicted in ​ is structured as a Clos network. A Clos network is a particular type of network topology that is designed to provide connection redundancy while maintaining high bi-section bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-stage or multi-level switching network, where the number of stages or levels can be two, three, four, five, etc. ​ The embodiment depicted in ​ is a 3-level network, including levels 1, 2, and 3. The TOR switch 504 represents the level-0 switch in the Clos network. One or more NVDs are connected to the TOR switch. The level-0 switch is also referred to as the edge device of the physical network. The level-0 switch is connected to the level-1 switches, which are also referred to as leaf switches. In ​In the illustrated embodiment, a set of "n" tier-0 TOR switches are connected to a set of "n" tier-1 switches and together form a pod. Each tier-0 switch in the pod is interconnected to all of the tier-1 switches in the pod, but there is no switch connectivity between pods. In some embodiments, two pods are referred to as a block. Each block is served by or connected to a set of "n" tier-2 switches (sometimes referred to as spine switches). There can be several blocks in the physical network topology. The tier-2 switches are in turn connected to "n" tier-3 switches (sometimes referred to as super spine switches). Communication of packets on the physical network 500 is typically performed using one or more layer-3 communication protocols. Generally, all layers of the physical network (except the TOR layer) are n-way redundant, thus allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, enabling the physical network to scale.

[0164] A characteristic of a Clos network is that the maximum number of hops from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is fixed. For example, in a 3-tier Clos network, a packet takes at most seven hops to reach from one NVD to another NVD, where the source and destination NVDs are connected to the leaf tier of the Clos network. Similarly, in a 4-tier Clos network, a packet takes at most nine hops to reach from one NVD to another NVD, where the source and destination NVDs are connected to the leaf tier of the Clos network. Thus, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. The Clos topology scales horizontally and is cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches at each tier (e.g., more leaf switches and spine switches) and by increasing the number of links between switches in adjacent tiers.

[0165] In some embodiments, each resource within the CSPI is assigned a unique identifier, referred to as a cloud identifier (CID). This identifier is included as part of the information for the resource and can be used to manage the resource, e.g., via a console or through an API. An example syntax for a CID is:

[0166] ocid1.<RESOURCE TYPE>. <realm>.[REGION][.FUTURE USE]. <uniqueid>

[0167] Among them,

[0168] ocid1: A text string that indicates the version of the CID;

[0169] resource type: The type of the resource (e.g., instance, volume, VCN, subnet, user, group, etc.);

[0170] realm: The realm where the resource is located. Example values are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal government cloud realm, etc. Each realm can have its own domain name;

[0171] region: The region where the resource is located. If this region is not applicable to the resource, then this part can be empty;

[0172] future use: Reserved for future use.

[0173] uniqueID: The unique part of the ID. The format can vary depending on the type of the resource or service.

[0174] ​

[0175] For the purposes of this application, a virtual network interface card (VNIC) provides a virtual network interface for a compute instance associated with the VNIC, enabling the compute instance to become part of or connect to a virtual cloud network (VCN). The VNIC can be implemented or executed by the NVD. When a customer launches a compute instance on a server or host machine that contains a NIC, the instance uses the networking service VNIC to communicate. The VNIC enables the instance to connect to a virtual cloud network (VCN) and determines how the instance connects to endpoints inside and outside the VCN. Each VNIC resides in a subnet within the VCN. The VNIC can include the primary private IPv4 address from the subnet where the VNIC is located. If an IPv6 prefix is assigned to the subnet, then the primary IP address can be an IPv6 address. The VNIC can also include a MAC address, a VLAN tag, a flag for enabling or disabling source / destination checks for network traffic of the VNIC, etc. In some embodiments, a compute instance can have multiple associated VNICs, allowing the compute instance to participate in multiple different VCNs. In certain implementations, each VNIC is considered a resource (e.g., a logical resource) within the CSP's infrastructure and is associated with a unique resource ID (e.g., an Oracle Cloud ID within Oracle Cloud Infrastructure). Each compute instance has a primary VNIC, which is automatically created and attached when the compute instance is created and launched. The primary VNIC is associated with an IP address that resides in the subnet specified by the customer during launch.

[0176] An NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device having one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline, etc.), memory including a cache, and ports. Various virtualization functions can be performed by software / firmware executed by one or more processing units of the NVD. The NVD can be implemented in various different forms. For example, in some embodiments, the NVD is implemented as an interface card, which is referred to as a smartNIC, or a smart NIC with an on-board embedded processor. A smartNIC is a device separate from the server or host machine that executes a computing instance and is also separate from the physical NIC on the server or host machine.

[0177] The NVD can take various forms and implementations. In some implementations, the NVD can include multiple physical ports that enable the NVD to be connected to one or more host machines and to one or more TOR switches. The NVD can receive packets (e.g., packets and frames generated by a computing instance hosted by the host machine) from the host machine via the host-facing ports and, after performing the necessary packet processing, can forward the packets to the TOR switch via the network-facing ports of the NVD. The NVD can receive packets from the TOR switch via the network-facing ports of the NVD and, after performing the necessary packet processing, can forward the packets to the host machine via the host-facing ports of the NVD. In the present disclosure, the NVD and the smartNIC can be used interchangeably.

[0178] ​ is a block diagram illustrating a distributed environment 600 that includes an exemplary overlay network DDoS mitigation system (ONDMS) according to some embodiments. ​ The distributed environment 600 including the ONDMS depicted in is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment 600 can have more or fewer systems or components than those shown in ​ can combine two or more systems, or can have different configurations or arrangements of the systems. ​ The systems, subsystems, and other components depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device).

[0179] As ​ As shown in, the distributed environment 600 includes a CSPI 601 and one or more senders 602a-n. A sender is a source of network traffic, including packets. Some senders may be external to the CSPI, and some may be internal to the CSPI. The CSPI may include (1) receivers, such as computing instances hosted by a host machine 650, for receiving network traffic from the senders; (2) Receiver-NVDs (or receiving NVDs), such as NVD-Rx 620, which act as routing devices and are located between the senders and the receivers; (3) an ONDMS 690 for performing DDoS attack mitigation; and (4) a control plane (e.g., a VCN control plane) 640.

[0180] In some embodiments, a sender may send packets to a receiver via two different paths. The first path is for senders internal to the CSPI. For example, a computing instance on a sender machine (e.g., 602a) may issue a packet, pass through the sender's NVD (e.g., 604a), reach the receiver's NVD (e.g., 620), and then arrive at the host machine 650 hosting the computing instance, i.e., the ultimate receiver. The second path is for senders external to the CSPI. For example, a computing instance on a sender machine (e.g., 602n) may issue a packet, pass through the sender's gateway (e.g., 604n), reach the receiver's NVD (e.g., 620), and then arrive at the host machine 650 hosting the computing instance, i.e., the ultimate receiver.

[0181] In ​ which, 604a-n are NVDs that can perform transmission and reception functions. In ​ the context of DDoS mitigation depicted in, 604a-n perform the transmission function of sending packets to receivers in the CSPI and may be referred to as NVD-Tx1, NVD-Tx2, etc. The VNICs implemented by the NVDs 604a-n may also be referred to as VNIC-Tx1, VNIC-Tx2, etc. Similarly, 620 and 621a-n are NVDs that can perform transmission and reception functions. However, in ​ the context of DDoS mitigation described in, 620 performs the reception function of receiving packets from the senders 602a-n and may be referred to as NVD-Rx or receiver NVD. The VNICs implemented by the NVD 620 may also be referred to as VNIC-Rx1, VNIC-Rx2, etc.

[0182] In ​ In the transmission side, as discussed above, some senders may be located outside the CSPI, and some senders may be located inside the CSPI. For those external senders, they use gateways to communicate with components inside the CSPI. For example, senders 602a and 602b are inside the CSPI, while sender 602n is outside the CSPI. An NVD (e.g., NVD-Tx) or a gateway can be associated with each sender. As an example, in ​ a sender (e.g., 602a) can be in the same VCN as the receiver-NVD 620. Thus, the transmission NVD (NVD-Tx1) 604a can be associated with sender 602a. However, the sender (e.g., 602n) is outside the CSPI where the receiver-NVD 620 is located. Thus, the Internet gateway (IGW) 604n can be associated with sender 602n.

[0183] The gateway can be a virtual interface configured for the VCN and enable the transfer of traffic to and from one or more endpoints outside the VCN. Gateway 604n can include different types of gateways, such as a dynamic routing gateway (DRG) for facilitating traffic transfer between the cloud infrastructure and the customer on-premises network, and an Internet gateway (IGW) for facilitating traffic transfer between the cloud infrastructure and a public network (such as the Internet).

[0184] Different technologies can be used to implement gateway 604n. The gateway can be implemented using only software, using hardware, or a combination of software and hardware. The gateway can be implemented as a logical or virtual or overlay network construct, such as a virtual router executed by a host machine or server, a Linux device, an NVD, etc. The gateway can also be implemented as a physical network device, such as a physical router. The gateway can be dynamically configured as needed.

[0185] On the receiving side, a receiver-NVD containing VNIC-Rx1 622a is illustrated, namely NVD-Rx (e.g., smartNIC) 620. In some embodiments, NVD-Rx 620 can contain more than one receiver-NVD (VNIC-Rx), which can be tagged as 622a, 622b, …, 622n (not shown). NVD-Rx 620 also includes a usage information collector (UIC) 624 with a telemetry service that transfers the collected usage information to a distributed denial of service (DDoS) monitor 642. More details about the UIC are described below.

[0186] In ​ Among them, an overlay network DDoS mitigation system (ONDMS) 690 for performing DDoS mitigation in an overlay network in CSPI may include a DDoS scrubber system 662 and a DDoS monitoring service, and the DDoS monitoring service includes a DDoS monitor 642 and a plurality of usage information collectors (UICs), such as UIC 624 in the receiver-NVD 620 and UIC 664 in the DDoS scrubber system 662. In some embodiments, the DDoS monitoring service may monitor network traffic (e.g., packets) received by the receiver-NVD (e.g., 624), and the receiver-NVD may forward the packets to the receiver. When the DDoS monitoring service determines that a potential DDoS attack has occurred, the network traffic may be redirected to the DDoS scrubber system 662.

[0187] In certain embodiments, the DDoS scrubber system 662 may be configured to analyze the redirected packets and take appropriate mitigation actions (e.g., filtering and throttling). The DDoS scrubber system 662 includes a set of host machines or NVDs (e.g., at least one host machine / NVD or a team of host machines / NVDs), and these host machines or NVDs are configured to implement one or more shadow VNICs (e.g., SD-VNIC1 660a, and additional SD-VNICs tagged as 660b,..., 660n 2-N , ​ not shown in the figure). The shadow VNIC may be implemented by one or more systems of devices different from the receiver-NVD (e.g., NVD-Rx 620).

[0188] In some embodiments, the TOR switch (e.g., 680) may be connected to many receiver-NVDs (e.g., 612a to 612n) in addition to being connected to the NVD-Rx620. Therefore, the packets sent to the receiver 650 may first reach the TOR switch 680, and the TOR switch 680 forwards the packets to different receiver NVDs, including the receiver-NVD 620 associated with the receiver 650.

[0189] In some embodiments, the ONDMS may include a DDoS monitoring service that includes a UIC 624 in a Receiver-NVD 620, a UIC 664 in a DDoS scrubber system 662, and a DDoS monitor 642. The UIC 624 may monitor and collect resource utilization information (including network usage, bandwidth, and utilization information 641) of network devices (or network resources) in a Receiver-NVD (NVD-Rx) 620 (e.g., a smartNIC), while the UIC 664 may monitor and collect network usage, bandwidth, and utilization information 641 of one or more shadow VNICs (SD-VNICs) in the DDoS scrubber system 662. The DDoS monitor 642 works with both the UIC 624 and the UIC 664 to analyze the collected network information related to various network devices. The collected information may include, but is not limited to, CPU utilization information of the Receiver-NVD 620, information of an individual receive VNIC (e.g., VNIC-Rx1 622a) executed by the Receiver-NVD 620, bandwidth utilization information of a smartNIC link (towards the host machine and to the TOR), etc. The types of information collected may include, but are not limited to, packet rate (i.e., packets per second (PPS)) information, bandwidth-related information, traffic patterns, and trends. In some embodiments, the DDoS monitor 642 may be part of the control plane 640. In other embodiments, the DDoS monitor 642 may be an entity separate from the control plane 640.

[0190] In some embodiments, the UIC 624 may be placed in many network locations, including the TOR, because large DDoS metrics (e.g., high or heavy traffic) may not be observable at the NVD level. For example, heavy aggregated traffic may occur at the TOR level but not at the individual NVD level. The UIC 624 may also monitor the traffic sent from all senders to the VNICs on the Receiver-NVD 620 and aggregate the traffic at the smartNIC level. In some embodiments, the UIC 624 may be placed outside the NVD (e.g., a smartNIC). For example, to protect the Receiver-NVD (e.g., 620), the UIC 624 may monitor the links between the NVD and the top-of-rack (TOR) and between the NVD and the host machine. The monitored and collected traffic (and usage) information is sent to the DDoS monitor to determine whether the traffic meets the requirements and thresholds of a DoS.

[0191] Another reason for placing UIC 624 in multiple network locations can benefit the DDoS monitoring service is such a scenario where, if the NVD is receiving too much data (e.g., packets with a large amount of data), even if the packet rate (or PPS) is still within the limit range. This scenario makes it difficult to measure whether there is excessive traffic and report it to the DDoS monitor. Therefore, DDoS attacks are difficult to detect. In such a scenario, the DDoS monitoring service can rely on the metrics collected from various locations to piece together the overall picture of the NVD and VNIC receiving.

[0192] In some embodiments, the DDoS monitoring service performs fine-grained utilization measurements on the packet rate (or PPS) and bandwidth utilization. The measurements are also performed at short intervals to capture microbursts, as opposed to significant congestion events that cannot be captured by the one-minute average. The DDoS monitoring service also has bandwidth metrics at the TOR, because sometimes at the NVD level, the NVD does not know that it is receiving traffic higher than it can receive. Therefore, the TOR can detect this situation due to, for example, backpressure. As an illustration, the link at the TOR switch (e.g., 626_tor) is 100 Gbps. The NVD (e.g., link 626_tor) may not know about network traffic exceeding 100 Gbps, but the TOR does because packets are dropped at the TOR. The limited bandwidth between the rack and the TOR can still lead to the noisy neighbor problem.

[0193] The DDoS monitoring service makes decisions to redirect (or remap) traffic based on the metrics collected (considering the sender, receiver, and TOR). When thresholds at different locations are reached, the monitoring service can send a message or trigger an event (referred to herein as a DDoS event) in the CP (e.g., VCN CP) to add protection to a specific VNIC or all VNICs implemented by the NVD, and the protection condition is referred to herein as the protected mode. Detection of a DDoS attack can be based on aggregated received information (e.g., each sending instance is within the traffic limit, but the aggregated traffic at the receiving end exceeds the traffic limit). In some embodiments, some predetermined thresholds (also referred to as DDoS criteria 643) can be used to trigger a DDoS event (e.g., a potential DDoS attack) (i.e., a type of network congestion event). For example, when network congestion occurs on the receiving link 626a of the protected VNIC 622a of the receiver-NVD 620 with a link utilization burst of 100% or higher for several consecutive minutes (e.g., three to five minutes), or when the average link utilization on the receiving link 626a is greater than 80% for five to ten consecutive minutes, a DDoS event can be triggered. In some embodiments, the duration of the congestion burst or average link utilization may be shorter or longer, depending on the application and the need for DDoS event sensitivity. The DDoS monitoring service that monitors incoming, outgoing, and traffic patterns or trends, running on or outside the NVD, uses the telemetry service to convey information to the VCN CP.

[0194] Reference ​ , after the DDoS monitor 642 of the DDoS monitoring service determines that a DDoS event has been triggered (i.e., the DDoS criteria are met), the DDoS monitor 642 notifies the CP 640 via the API 644 to enter the protected mode. The protected mode can be used to (1) protect an individual receiving VNIC 622a on the receiver-NVD, or (2) protect the receiver-NVD 620.

[0195] In some embodiments, the shadow VNIC generator 645 in the control plane 640 can receive the protected mode notification from the DDoS monitor 642 to perform the process for creating shadow VNICs. In the case of protecting the receiver-NVD (e.g., 620), the shadow VNIC generator 645 can determine all VNICs or some selected VNICs implemented by the NVD to be protected, and then create shadow VNICs for these VNICs. Then, the shadow VNIC generator 645 can send a signal to the distribution service 646 to publish the VNIC mapping.

[0196] For individual VNIC protection, in some embodiments, the NVD 620 identifies the receiving VNICs of the NVD to be protected due to excessive VNIC traffic and notifies the VCN CP to create shadow VNICs (SD-VNICs) for the receiving VNICs (referred to herein as protected VNICs). The ONDMS can redirect the traffic received by the protected VNICs to the newly created SD-VNICs. For example, if the receiving VNIC (VNIC-Rx1 622a) is selected for protection, then the shadow VNIC (SD-VNIC1 660a) can be created by the CP 640. If additional VNIC-Rx 622b-n (not shown) in the NVD 620 are selected for protection, then other SD-VNICs 660b-n (not shown) can be created by the CP 640. For example, if another receiving VNIC (e.g., VNIC-Rx2 622b not shown) is selected for protection, then an additional SD-VNIC (e.g., SD-VNIC2 660b) is created for protecting the VNIC-Rx2 622b.

[0197] In some implementations, the ONDMS can track the VNICs to be protected by storing the resource identifier (ID) corresponding to the VNICs to be protected in a repository (such as a memory or a database) of the CP 640. Generally, a unique resource identifier (resource ID) is used to identify resources within the cloud infrastructure. When a resource is created, a resource ID can be assigned to the resource. When the UICs 624 and 664 collect usage information, the resource ID of the VNIC can be tagged with the usage information and sent to the DDoS monitor 642. The DDoS monitor can look up the stored resource ID in the repository to determine whether any of the protected VNICs receive excessive traffic.

[0198] For NVD protection, in some embodiments, due to excessive traffic at the NVD level, the entire receiver-NVD 620 can be protected, including VNIC-Rx 622a-n (if there are multiple receiving VNICs in the receiver-NVD 620). In other words, an equal number of shadow VNICs (SD-VNIC 660a-n) can be created to protect the receiving VNICs corresponding to these shadow VNICs, VNIC-Rx 622a-n (i.e., a one-to-one mapping between the SD-VNIC and its protected VNIC, or one SD-VNIC for each VNIC). For example, the SD-VNIC1 660a is created to protect the VNIC-Rx1 622a, the SD-VNIC2 660b (not shown) is created to protect the VNIC-Rx2 622b (not shown), and the SD-VNICn 660n (not shown) is created to protect the VNIC-Rxn 622n (not shown).

[0199] In some alternative embodiments, a single SD-VNIC can be created for multiple protected VNICs (i.e., a one-to-many mapping between the SD-VNIC and its protected VNICs). For example, during individual VNIC protection, as discussed above, an SD-VNIC is created in the NVD for the protected VNIC. If an additional receiving VNIC is to be protected, then this second VNIC can also be protected by the same SD-VNIC. In other words, the number of SD-VNICs can be less than the total number of protected VNICs in the NVD. For further illustration, for NVD protection, all receiving VNICs on the NVD can be protected by a single SD-VNIC. For example, VNIC-Rx 622a-n can be protected by a single SD-VNIC1 660a.

[0200] In some embodiments, certain network configurations (such as IP and routing and security configurations) can be the same between the SD-VNIC and the protected VNIC. For example, the SD-VNIC can have the same IP configuration (e.g., DHCP or static), routing configuration (e.g., policy-based or dynamic routing), and security configuration (e.g., firewall and access control) as the protected VNIC (i.e., the original destination). The SD-VNIC (e.g., 660a) can be a service VNIC that is invisible to the customer but is hosted (or implemented) by a dedicated DDoS scrubber system 662 capable of performing DDoS filtering and / or throttling. In some embodiments, the SD-VNIC can be separately executed by an individual host on the network link between the sender and the protected VNIC of the NVD. The SD-VNIC can be regarded as a copy of the original protected VNIC (e.g., 622a) in the NVD (e.g., 620), and this SD-VNIC receives packets from the senders 602a-n. For example, after the DDoS monitoring service detects a DDoS event, the traffic is redirected from the protected VNIC to the SD-VNIC. The SD-VNIC has a configuration similar to that of the protected VNIC and can check security rules, but does not directly process the packets and send the packets to the receiver 650 (e.g., one or more computing instances of the customer). Instead, the SD-VNIC (e.g., 660a) receives network traffic on behalf of the original protected VNIC (622a), so that the hosted DDoS scrubber system can perform throttling (i.e., intentionally controlling network traffic to regulate the flow of data packets) and / or filtering (i.e., selectively allowing or blocking specific types of network traffic based on defined filtering criteria or rules), which can include but are not limited to DDoS scrubbing and stateless security rule enforcement, and deliver the filtered packets to the original protected VNIC. The filtering can be performed by the DDoS scrubber system 662 based on certain filtering criteria 663.

[0201] DDoS attack mitigation or scrubbing can include layer 3 / 4 mitigation and layer 7 mitigation. For example, layer 3 / 4 mitigation can detect and mitigate attacks such as SYN / ACK floods, UDP reflection amplification, and DNS query floods. Layer 7 mitigation can use a web application firewall (WAF) service to detect and block malicious HTTP / HTTPS traffic.

[0202] Stateless security rule enforcement means that security rules are stateless. A stateless security rule means that connection tracking is not used for any traffic that matches the rule. Compared to stateful security rules that track the state of established connections and allow return traffic related to those connections, stateless security rules evaluate each packet independently, without considering any previous connections. Security rules are used to control traffic at the packet level and can include, but are not limited to, security lists and network security groups. A security list contains a set of ingress and egress rules that specify the types of traffic that are allowed and is applied to all VNICs in a given subnet. A network security group contains a set of ingress and egress security rules that apply only to a set of VNICs in a single VCN.

[0203] Because the SD-VNIC has a greater bandwidth than the protected VNIC, excessive traffic does not directly affect other customers. The SD-VNIC, for example, helps handle traffic and prioritize traffic by applying security rules to discard certain packets or throttle excessive traffic to meet the required limits of the protected VNIC. Thus, the SD-VNIC ensures that the traffic delivered to the protected VNIC is legitimate and within the allowed limits.

[0204] When the ONDMS enters the protected mode for a specific protected VNIC or NVD, the traffic received by the protected VNIC can be redirected to one or more shadow VNICs implemented by the DDoS scrubber system by updating the VNIC mapping. In some embodiments, the VNIC mapping can be an association between an overlay address and a base address. As discussed previously, in ​ the overlay address can be configured for a receiver (e.g., a compute instance) 650. The VNIC 622a that enables the receiver 650 to be part of the virtual cloud network can also be associated with the overlay address. The NVD 620 that implements the VNIC 622a can be configured with a base address within the CSPI. Similarly, the DDoS scrubber system 662 that implements the SD-VNIC 660a can be configured with a base address within the CSPI.

[0205] When the ONDMS redirects traffic received by a protected VNIC to a newly created SD-VNIC, the ONDMS can update the VNIC mapping stored in a repository (e.g., non-volatile memory, database, etc.) for a network component (such as an NVD or gateway associated with the traffic source, i.e., the sender). The repository for each network component (e.g., NVD or gateway) can be located either inside or outside the component. For example, the VNIC mapping can include a table that includes the overlay address of the destination receiver and the base address of the NVD to which that overlay address is mapped, such that a packet with the destination overlay address can be forwarded to the mapped base address in the CSPI. As an example, in normal operation mode, the destination overlay address can be the overlay IP address X of the receiver (compute instance) 650, and the base address to which that overlay address is mapped can be the base IP address X of the receiver-NVD 620 that implements the VNIC 622a associated with the receiver 650. After the VNIC mapping update 628 for entering the protected mode, the same overlay IP address X of the receiver (compute instance) 650 is mapped to a different base address, e.g., the base IP address Y of the DDoS scrubber system 662 that implements the shadow VNIC (SD-VNIC1 660a). In other words, the mapping update can look like the following:

[0206] Mapping for normal operation mode: (Overlay IP address X of the receiver, Base IP address X of the receiver-NVD)

[0207] Mapping for protected mode: (Overlay IP address X of the receiver, Base IP address Y of the DDoS scrubber system)

[0208] In some embodiments, as ​ As shown, the distribution service 646 of the CP 640 performs VNIC mapping updates. The distribution service 646 is a service that distributes new mappings to all NVDs (e.g., 604a and 604b) or gateways (e.g., 604n) associated with the sender depending on the location of the sender. For example, in one embodiment, if the sender 602a is in the same VCN as the receiver-NVD 620, then the packet can be sent from a different subnet through the local network. The NVD 604a associated with the sender can store the VNIC mapping, and the NVD 604a of the sender 1 602a only needs to look up the VNIC mapping table to find the destination of the received packet. In another embodiment, if the sender (e.g., 602n) is in a different VCN from the VCN where the receiver-NVD 620 is located, then the VNIC mapping table can be stored in the local peer gateway (LPG, e.g., 604n). In another embodiment, if the sender (e.g., 602n) is in a different region from the region where the receiver-NVD 620 is located, then the VNIC mapping table can be stored in the dynamic routing gateway (DRG, e.g., 604n). In yet another embodiment, if the sender (e.g., 602n) is outside the cloud infrastructure where the receiver-NVD 620 is located, then the VNIC mapping table can be stored in the internet gateway (IGW, e.g., 604n).

[0209] To perform in ​ Further described in, for individual VNIC protection, assume that before the VNIC mapping is updated, traffic 606a from sender-NVD 604a, traffic 606b from sender-NVD 604b, and traffic 606n from sender gateway 604n all go to the same VNIC-Rx1 622a of receiver-NVD 602. The received traffic 626a at VNIC-Rx1 622a monitored by the DDoS monitoring service may exceed a preset threshold, and cause the DDoS monitor 642 to notify the CP640 to enter the protected mode through the API 622. The distribution service 646 of the CP 640 can be configured to update the VNIC mapping stored in the NVD or gateway 604a-n through the control link 628 by changing the mapped base IP address from the receiver-NVD 620 to the DDoS scrubber system 662. Thus, the packets carrying traffic 606a-n are redirected to the SD-VNIC1660a through different routes 666a-n. In some embodiments, the new VNIC mapping may cause the NVD and gateway 604a-n to encapsulate the packet header with the destination address of the SD-VNIC1660a. The DDoS scrubber system 662 implementing the SD-VNIC1 660a can perform DDoS filtering and / or throttling, and forward the packet to the protected VNIC-Rx1 622a by removing the added / encapsulated packet header through the route 668_tor, and then 668a. The protected VNIC-Rx1 622a can then perform its normal packet processing and forward the processed packet to the receiver 650.

[0210] In some embodiments, as discussed above, a similar mechanism as described above is applied for NVD protection, and if there are multiple receiving VNICs in the NVD 620, the CP 640 may also create its corresponding additional SD-VNIC2s 660b to 660n (not shown) for the protected VNIC-Rx2 622b to VNIC-Rxn 622n (not shown). Packets from the sender will go to the routes 666a-n instead of 626a-n (where, respectively, 626b goes to VNIC-Rx2 622b, and 626n goes to VNIC-Rxn 622n, not shown). After the DDoS scrubber system 662 implementing the SD-VNICs 660a-n performs DDoS filtering and / or throttling, the packets are forwarded to the protected VNIC-Rx622a-n via the route 668_tor and then 668a-n (where, respectively, 668b goes to VNIC-Rx2 622b, and 668n goes to VNIC-Rxn 622n, not shown). The protected VNIC-Rx 622a-n in the NVD620 can then perform the normal packet processing for these VNICs and forward the processed packets to the receiver 650.

[0211] In some embodiments, in addition to performing filtering and / or throttling on the receiving ends 666a-n after detecting a DDoS event, the ONDMS may also notify the senders (e.g., 602a-n) to throttle the outgoing traffic of these senders.

[0212] In some embodiments, a UIC 664, similar to the UIC 624 of the Receiver-NVD 620, for a DDoS monitoring service is also available on the DDoS scrubber system 662, which monitors and collects traffic related to the SD-VNIC of an individual protected VNIC (e.g., 660a) or all SD-VNICs (e.g., 660a-n) of the Receiver-NVD 620 to collect traffic information and communicate with the DDoS monitor 642 to determine whether the collected information meets the DDoS criteria 643 for ending / quitting the protected mode. For example, when the network link utilization drops below a predetermined threshold within a predetermined period of time, e.g., the aggregated traffic to the protected VNIC has a link utilization below 80% for ten consecutive minutes (e.g., five to ten minutes), then the DDoS monitor 642 can notify the CP 640 that the protected mode exit condition is met. The DDoS monitor 642 works with both the UIC 624 and the UIC 664 to ensure a proper transition into and out of the protected mode, as the DDoS monitor has visibility into both the UIC 624 and 664. The CP 640 can delete the SD-VNIC 660a for the protected VNIC 622a and send a command to the distributed service 624 to update the VNIC mapping in the sender's NVD or gateways 604a-n. For example, the mapping update can look as follows:

[0213] Mapping before update (i.e., for protected mode): (Overlay IP address of the receiver, Base IP address of the DDoS scrubber system)

[0214] Mapping after update (i.e., for normal operation mode): (Overlay IP address of the receiver, Base IP address of the Receiver-NVD)

[0215] Then, the ONDMS can return to the normal operation mode. In some embodiments, the SD-VNICs in the DDoS scrubber system 662 may not be deleted and only the VNIC mapping is updated.

[0216] ​ FIG. 700 is a flowchart depicting a processing flow for a network resource to enter a protected mode according to certain embodiments. ​ The processing depicted in FIG. may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented in FIG. and described below are intended to be illustrative and not restrictive. While ​ depicts various processing steps that occur in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel.

[0217] As ​ depicted in, the process is initiated at 702. At 702, resource utilization information that can be monitored and collected for network resources such as receiving VNICs, receiving NVDs, and their associated TORs can be obtained. For example, ​ the UIC 624 of the NVD 620 in collects usage information for different levels of network resources such as VNIC level, NVD level, and TOR level. At 704, based on the information collected at 702, DDoS event conditions for the protected network resources can be determined based on the DDoS standard 643 set by the ONDMS. For example, ​ the DDoS monitor 642 in can determine whether the aggregated traffic received at the link 626a for the protected VNIC-Rx1 622a meets the conditions for a DDoS event, for example, as described in ​ the predetermined threshold discussed. If the DDoS event conditions for the receiving VNIC to be protected are not met, then the process goes to 702 and continues to monitor and collect network information. If the DDoS event conditions for the receiving VNIC to be protected are met, then the process goes to 706.

[0218] At 706, the network resources associated with the satisfied DDoS event conditions can be identified. The network resources can be VNICs (e.g., 622a), or they can be NVDs (e.g., 620) that implement one or more VNICs. For example, if the network resource is a VNIC, then when the aggregated received network traffic for the VNIC-Rx1 622a implemented (or executed) by the NVD 620 meets the DDoS event conditions, the VNIC-Rx1 622a can be identified by the CP 640 for protection based on the resource identification number (e.g., Oracle Cloud ID or OCID) of the VNIC-Rx1 622a. If the network resource is the receiver-NVD 620, then when the aggregated received network traffic for the receiver-NVD 620 at the NVD level meets the DDoS event conditions, these VNICs can be identified based on the resource identification numbers of all the VNICs (622a-n) implemented by the receiver NVD 620. In some embodiments, even if the network resource is an NVD, multiple (but not all) VNICs (i.e., a few selected VNICs) in the NVD can be identified for protection.

[0219] 708 can cover 710 to 716. At 708, ONDMS enters the protected mode for the network resources identified in 706. For example, in some embodiments, the DDoS monitoring service can enter the condition or specific state of the state machine indicating that the protected mode has been activated for the protected VNIC-Rx1622a.

[0220] At 710, the CP is notified about the specific network resources to be protected. For example, ​ the DDoS monitor 642 in [5] can notify the CP 640 about the VNIC to be protected by providing the corresponding resource ID of the specific VNIC 622a that meets the DDoS event condition for the aggregated received traffic. At 712, the CP (or the shadow VNIC generator 645 in the CP) creates one or more SD-VNICs for the network resources identified in 706. For example, if the network resource is a VNIC, then ​ the CP 640 in [7] can use the resource ID information of the identified VNIC (e.g., 622a) provided by the DDoS monitor 642 to request the cloud resource manager to allocate appropriate resources to create a shadow VNIC, SD-VNIC (e.g., 660a) for the VNIC-Rx1 622a.

[0221] For example, if the network resource is an NVD (e.g., 620), then ​ the CP 640 in

[12] can use the resource ID information of the identified VNIC (e.g., 622a-n) provided by the DDoS monitor 642 to request the cloud resource manager to allocate appropriate resources to create one or more SD-VNICs for the receiver-NVD 620, depending on whether a single SD-VNIC660a is used to protect the receiver-NVD (i.e., one-to-many mapping between the SD-VNIC and its selectively protected VNIC), or an equal number of SD-VNICs (660a-n) are used to protect all the VNICs (622a-n) implemented by the receiver-NVD 620 or the selected VNICs (i.e., one-to-one mapping).

[0222] At 714, the CP can publish information related to the one or more shadow VNICs created in 712, which causes the packets received by the network resources to be redirected to the DDoS scrubber system that implements the SD-VNIC. In other words, the VNIC mapping stored in the NVD and / or gateway associated with the source of the traffic received by the protected VNIC or NVD is updated by the CP or other entities in the VCN, so that the traffic received by the protected VNIC or NVD is redirected to the DDoS scrubber system. For example, in ​ In , the VNIC mapping (e.g., the mapping tables of the NVD and / or gateway 604a-n associated with the sender 602a-n) can be updated such that the SD-VNIC1 660a can replace the VNIC-Rx1 622a to become the new destination of the sender 602a-n. Thus, the packets received by the protected VNIC 622a implemented by the Receiver-NVD 620 can be redirected to the DDoS scrubber system 662 implementing the SD-VNIC 660a instead. For example, in ​ the transport packets from the sender 602a-n can go to the SD-VNIC1 660a via the route 666a-n instead of going to the VNIC-Rx1 622a via the route 626_tor and then 626a.

[0223] At 716, for the packets redirected to the DDoS scrubber system, the DDoS scrubber system performs actions (e.g., filtering, throttling, etc.) on the packets, and these actions control which packets will be forwarded to the network resources being protected. In other words, during the protected mode, the DDoS scrubber system 662 implementing one or more SD-VNICs performs DDoS scrubbing, security rule checking, and / or throttling on the redirected packets received by the DDoS scrubber system and then forwards them to the protected VNIC-Rx1 622a or the Receiver-NVD 620.

[0224] ​ is a flowchart illustrating a processing flow for a network resource to exit the protected mode according to some embodiments. ​ The processing depicted in can be implemented by software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented in and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps occurring in a particular order or sequence, this is not intended to be limiting. In some alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0225] In ​ for a network resource in the protected mode, at 718, monitoring and collection of resource utilization information for one or more shadow VNICs protecting the network resource can continue. Since the ONDMS is in the protected mode and the packets are redirected to the DDoS scrubber system, the resource utilization information for the protected network resource is observed at one or more receiving SD-VNICs. For example, ​ The UIC 664 therein can collect the resource utilization or network usage information of the SD-VNIC1 660a that protects the VNIC 622a and transmit the status of the network during the protected mode to the DDoS monitor 642.

[0226] At 720, based on the information collected in 718, the protected mode exit condition for the protected VNIC can be determined based on the DDoS standard set by the ONDMS, as discussed above with respect to ​ If the protected mode exit condition is not met, the process goes to 718 and continues to monitor and collect resource utilization information. If the protected mode exit condition is met, the process goes to 722.

[0227] 722 can cover 724 to 728. At 722, the ONDMS can exit the protected mode for the network resources and return to the normal mode. For example, in some embodiments, in ​ the DDoS monitor 642 of the DDoS monitoring service can enter the condition or state where the indication of the state machine has deactivated the protected mode for the protected VNIC-Rx1 622a. At 724, the CP can be notified of the change in the protected mode exit condition and mode. For example, in ​ the DDoS monitor 642 can notify the CP 640 that the protected mode exit condition is met and it is ready to exit the protected mode for the protected VNIC-Rx1 622a by providing the corresponding resource ID of the protected VNIC-Rx1 622a.

[0228] At 726, one or more SD-VNICs created for the network resources can be deleted or removed to redirect the traffic back to the protected network resources. For example, in some embodiments, ​ the shadow VNIC generator 645 in the CP 640 in

[0229] At 728, the CP can republish the information (e.g., network resources) such that the packets received by the network resources but being redirected to the DDoS scrubber are now received by the network resources and are no longer redirected. In other words, the CP (e.g., the VCN CP or other entities in the VCN) updates the VNIC mapping stored in the NVD and / or gateway associated with the source of the traffic again to reflect the change in the mode, so that the traffic redirected to the DDoS scrubber is received by the protected network resources again. For example, in ​ In [the above], the VNIC mapping (such as a mapping table) in the NVD and / or gateway 604a-n associated with the sender 602a-n can be updated so that the base IP address of the DDoS scrubber system is removed and replaced by the base IP address of the receiver-NVD 620. In other words, the mapping changes from the protected mode to the normal operation mode. Then, this process is repeated and the flow proceeds to 702.

[0230] ​ FIG. 800 is a flowchart illustrating a process performed for an individual VNIC in the protected mode according to some embodiments. ​ The process depicted in [the above] can be implemented in software (such as code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented in [the above] and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps that occur in a particular order or sequence, this is not intended to be limiting. In some alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0231] At 802, the network resource to be protected is determined as a specific VNIC to be protected. For example, in ​ [the above], VNIC1 (or VNIC-Rx1) 622a implemented by the NVD 620 can be identified for protection based on the resource identification number of the VNIC.

[0232] At 804, the CP can be notified that the particular VNIC is going to be protected. For example, ​ the DDoS monitor 642 in [the above] can notify the CP 640 about the particular VNIC by providing the corresponding resource ID of the particular VNIC 622a for which the aggregated received traffic meets the DDoS event condition.

[0233] At 808, the CP can create a shadow VNIC for the specific VNIC identified at 802. For example, ​ the CP 640 in [the above] can use the resource ID information of the identified VNIC (e.g., 622a) provided by the DDoS monitor 642 to request the cloud resource manager to allocate appropriate resources to create a shadow VNIC, SD-VNIC (e.g., 660a) for the VNIC-Rx1 622a.

[0234] At 810, the CP can publish information related to the shadow VNIC created in 808, which causes packets received by the specific VNIC to be protected to be redirected to the DDoS scrubber system implementing the shadow VNIC created in 808. For example, in ​ the VNIC mapping tables of the NVDs associated with senders 602a-n and / or gateways 604a-n can be updated such that the mapping for the normal operation mode (overlay IP address of the receiver, base IP address of the receiver-NVD) is updated to be the mapping for the protected mode (overlay IP address of the receiver, base IP address of the DDoS scrubber system). Thus, packets received by the protected VNIC 622a implemented by the receiver-NVD 620 can instead be redirected to the DDoS scrubber system 662 implementing the SD-VNIC 660a.

[0235] At 812, for packets redirected to the DDoS scrubber system, the DDoS scrubber system performs actions (e.g., filtering, throttling, etc.) on the packet that control which packets will be forwarded to the network resource being protected. In other words, during the protected mode, the DDoS scrubber system 662 implementing the SD-VNIC 660a can perform DDoS scrubbing, security rule checking, and / or throttling on the redirected packets received by the DDoS scrubbing system before forwarding them to the protected VNIC-Rx1 622a.

[0236] ​ is a flowchart 900 illustrating the processing performed for an NVD (e.g., smartNIC) in the protected mode according to certain embodiments. ​ The processing depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented in and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps occurring in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0237] As ​ depicted, the processing is initiated at 902. The network resource to be protected is determined as the specific NVD to be protected. For example, in ​ In this case, since the network resource to be protected is the Receiver-NVD 620, these VNICs can be identified based on the resource identification numbers of all the VNICs (622a-n) implemented by the Receiver-NVD 620. In some embodiments, several selected VNICs in the NVD can be identified for protection instead of all the VNICs implemented by the NVD.

[0238] At 904, the CP is notified about the specific network resource to be protected. For example, ​ the DDoS monitor 642 in can notify the CP 640 about the specific Receiver-NVD 620 to be protected where the aggregated received traffic meets the DDoS event condition.

[0239] At 906, the CP can determine one or more VNICs to be protected implemented by the NVD. For example, when the CP creates the Receiver-NVD 620, the CP may have information about which VNICs implemented by the NVD should be protected in the case of a DDoS attack. The resource IDs of these selected / identified VNICs to be protected in the Receiver-NVD 620 can be obtained.

[0240] At 908, the CP creates one or more shadow VNICs for the one or more VNICs identified in 906. For example, ​ the CP 640 in can use the resource IDs of the selected / identified VNICs to be protected to request the cloud resource manager to allocate appropriate resources to create one or more SD-VNICs for the identified VNICs. In some embodiments, a single SD-VNIC 660a can be created to protect the identified VNICs, as ​ discussed in the one-to-many mapping relationship. In other embodiments, an SD-VNIC can be created to protect each identified VNIC, as ​ discussed in the one-to-one mapping relationship.

[0241] At 910, the CP can publish information related to the one or more shadow VNICs created in 908, which causes the packets received by the NVD to be redirected to the DDoS scrubber system that implements the shadow VNICs created in 908. For example, in ​ In , the NVD associated with the sender 602a-n and / or the VNIC mapping table of the gateway 604a-n can be updated such that the mapping for the normal operation mode (the overlay IP address of the receiver, the base IP address of the receiver-NVD) is updated to the mapping for the protected mode (the overlay IP address of the receiver, the base IP address of the DDoS scrubber system). As a result, the packets received by the receiver-NVD 620 are redirected to the DDoS scrubber system 662 that implements one or more shadow VNICs.

[0242] At 912, for the packets redirected to the DDoS scrubber system, the DDoS scrubber system performs actions (e.g., filtering, throttling, etc.) on the packets, which control which packets will be forwarded to the NVD being protected. In other words, during the protected mode, the DDoS scrubber system 662 that implements one or more SD-VNICs performs DDoS scrubbing, security rule checking, and / or throttling on the redirected packets received by the DDoS scrubber system and then forwards them to the protected receiver-NVD 620.

[0243] ​ FIG. 1000 is a flowchart illustrating packet redirection for an individual VNIC in the protected mode according to certain embodiments. ​ The processes depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented in and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps occurring in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0244] ​ Further illustrates ​ the details in 716 of. At 1002, the sender issues a packet whose destination is the receiver entity associated with the protected VNIC. For example, in ​ , one of the senders 602a-n (e.g., 602a) can issue a packet that is sent to a receiver (e.g., a compute instance) 650 associated with the protected VNIC-Rx1622a performed by the NVD 620. The packet can be sent through one of the network links 606a-n (e.g., 606a), then 626_tor and 626a, to reach 652.

[0245] At 1004, based on the updated VNIC mapping, during protected mode, instead of packets being delivered to the NVD implementing the protected VNIC, the packets are redirected to the DDoS scrubber system implementing the shadow VNIC corresponding to the protected VNIC. For example, in ​ , the updated VNIC mapping in the NVD 604a associated with the sender 602a causes the packets on the link 606a to be redirected to the network link 666a to reach the SD-VNIC1 660a, rather than reaching the protected VNIC-Rx1 622a via the network links 626_tor and 626a. At 1006, the packets are received by the DDoS scrubber system implementing the SD-VNIC. For example, in ​ , after the VNIC mapping update, the SD-VNIC 660a (instead of the protected VNIC 622a) created for the protected VNIC 622a receives the packets.

[0246] At 1008, the DDoS scrubber system performs actions (such as filtering, throttling, etc.) on the received packets. 908 can also be broken down into sub-steps 1010 to 1017. At 1010, the DDoS scrubber system determines whether to forward the received packets to the NVD implementing the protected VNIC. For example, the DDoS scrubber system can perform throttling and / or filtering by discarding certain packets to manage bandwidth usage, reduce congestion, or enforce policies regarding data transfer rates. The DDoS scrubber system can also perform DDoS scrubbing and stateless security rule enforcement to filter packets, as discussed with respect to ​ .

[0247] At 1012, if the packet is filtered, then the packet can be discarded at 1016. If the packet is not filtered, then the process proceeds to 1013. At 1013, if the packet is to be throttled, then the DDoS scrubber system performs throttling at 1017. If the packet is not throttled, then the process proceeds to 1014. At 1014, the unfiltered or unthrottled packets are forwarded to the NVD implementing the protected VNIC. For example, in ​ , the DDoS scrubber system 662 implementing the SD-VNIC1 660a can forward the allowed packets after throttling and / or filtering to the receiver-NVD 620 implementing the protected VNIC 622a, and the receiver-NVD 620 can then forward the packets to the receiver, such as the compute instance 650.

[0248] ​ FIG. 1600 is a flowchart illustrating a generalized processing flow for the protected mode of a network virtualization device (NVD, e.g., smartNIC) according to certain embodiments. ​ The processes depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). ​ The methods presented in and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps that occur in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0249] As ​ depicted in, the process is initiated at 1602. Network traffic received by the NVD in the CSPI is monitored. The NVD can execute a set of one or more VNICs associated with a set of one or more compute instances in one or more overlay networks provided by the CSPI. The network traffic can be destined for at least one compute instance from the set of one or more compute instances.

[0250] At 1604, based at least in part on the monitoring, a protected mode can be initiated for the NVD to protect the NVD from potential DDoS attacks.

[0251] At 1606, when the NVD is in the protected mode, one or more packets destined for the set of one or more compute instances can be redirected to a DDoS scrubber system instead of being sent to the NVD.

[0252] ​ is a flowchart 1700 illustrating a generalized processing flow of the protected mode for an individual VNIC according to certain embodiments. ​ The processes depicted in can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). ​ The methods presented in and described below are intended to be illustrative and not restrictive. Although ​ depicts various processing steps that occur in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0253] As ​ As depicted, the process is initiated at 1702. Network traffic received by a first VNIC associated with a first computing instance in an overlay network provided by CSPI can be monitored. The network traffic can be destined for the first computing instance. The first VNIC can be associated with a first overlay address configured for the first computing instance.

[0254] At 1704, based at least in part on the monitoring, a protected mode can be initiated for the first VINC to protect the first VINC from potential DDoS attacks.

[0255] At 1706, when the first VINC is in the protected mode, one or more packets destined for the first computing instance can be redirected to a DDoS scrubber system rather than being sent to a first NVD that implements the first VNIC. The first overlay address can be associated with a base address associated with the NVD.

[0256] ​ FIG. 1800 is a flow chart illustrating a generalized processing flow for a protected mode for network resources according to certain embodiments. ​ The process depicted can be implemented by software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). ​ The methods presented herein and described below are intended to be illustrative and not restrictive. While ​ various processing steps are depicted as occurring in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, these steps can be performed in some different order, or some steps can also be performed in parallel.

[0257] As ​ depicted, the process is initiated at 1802. Network traffic received by multiple network resources in one or more overlay networks provided by CSPI can be monitored. The network traffic can be destined for computing instances.

[0258] At 1804, based at least in part on the monitoring, a protected mode can be initiated for a first network resource from among the multiple network resources to protect the first network resource from potential DDoS attacks. The first network resource can be associated with a first computing instance.

[0259] At 1806, when the first network resource is in the protected mode, one or more packets destined for the first computing instance can be redirected to a DDoS scrubber system rather than being sent to the first network resource.

[0260] ​

[0261] As pointed out above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also supply various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Thus, since these services can be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain the availability and performance of applications.

[0262] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of an application stack. For example, a user can log in to an IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into that VM. Then, the customer can use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, etc.

[0263] In most cases, the cloud computing model will require the participation of a cloud provider. The cloud provider can be but is not necessarily a third-party service that specifically provides (e.g., supplies, rents, sells) IaaS. An entity may also choose to deploy a private cloud and thus become its own infrastructure service provider.

[0264] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It can also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Thus, the customer can be responsible for handling the (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be launched on demand, etc.).

[0265] In some examples, IaaS provisioning can refer to obtaining a computer or virtual host for use and even installing the required libraries or services on that computer or virtual host. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0266] In some cases, there are two different challenges with IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is running. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.) after everything has been provisioned. In some cases, these two challenges can be addressed by enabling the configuration of the infrastructure to be defined in a declarative manner. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, after the topology has been defined, a workflow can be generated to create and / or manage the different components described in the configuration files.

[0267] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shared computing resources), also referred to as the core network. In some examples, one or more inbound / outbound traffic group rules can also be provisioned to define how the network's inbound / outbound traffic will be set up and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., can also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve incrementally.

[0268] In some cases, continuous deployment techniques can be employed to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the techniques described can enable the management of the infrastructure within these environments. In some examples, a service team can write code that is desired to be deployed to one or more but typically many different production environments (e.g., across various different geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, provisioning tools can be used to provision resources, and / or deployment tools can be used to deploy the code after the infrastructure has been provisioned.

[0269] ​ FIG. 1100 is a block diagram illustrating an example schema of an IaaS architecture according to at least one embodiment. A service operator 1102 can be communicatively coupled to a secure host lease 1104 that can include a virtual cloud network (VCN) 1106 and a secure host subnet 1108. In some examples, the service operator 1102 can use one or more client computing devices, which can be portable handheld devices (e.g., cellular phones, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google a head-mounted display), and the client computing device runs software (such as Microsoft Windows ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and enables Internet, email, Short Message Service (SMS), or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, a personal computer and / or a laptop computer running various versions of Microsoft Apple and / or Linux operating systems. The client computing device can be a workstation computer running any one of various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems (such as, for example, Google Chrome OS)). Alternatively or additionally, the client computing device can be any other electronic device capable of communicating via a network that can access VCN 1106 and / or the Internet, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a gesture input device), and / or a personal messaging device.

[0270] VCN 1106 can include a Local Peer Gateway (LPG) 1110, which can be communicatively coupled to a Secure Shell (SSH) VCN 1112 via the LPG 1110 included in the SSH VCN 1112. The SSH VCN 1112 can include an SSH subnet 1114, and the SSH VCN 1112 can be communicatively coupled to a control plane VCN 1116 via the LPG 1110 included in the control plane VCN 1116. In addition, the SSH VCN 1112 can be communicatively coupled to a data plane VCN 1118 via the LPG 1110. The control plane VCN 1116 and the data plane VCN 1118 can be included in a service lease 1119 that can be owned and / or operated by an IaaS provider.

[0271] The control plane VCN 1116 may include a control plane demilitarized zone (DMZ) layer 1120 that serves as an edge network (e.g., a portion of a corporate network between an intranet and an external network). The DMZ-based servers may have limited responsibilities and help control vulnerabilities. Additionally, the DMZ layer 1120 may include one or more load balancer (LB) subnets 1122, a control plane application layer 1124 that may include one or more application subnets 1126, and a control plane data layer 1128 that may include one or more database (DB) subnets 1130 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). The one or more LB subnets 1122 included in the control plane DMZ layer 1120 may be communicatively coupled to the one or more application subnets 1126 included in the control plane application layer 1124 and an Internet gateway 1134 that may be included in the control plane VCN 1116, and the one or more application subnets 1126 may be communicatively coupled to the one or more DB subnets 1130 included in the control plane data layer 1128, as well as a service gateway 1136 and a network address translation (NAT) gateway 1138. The control plane VCN 1116 may include a service gateway 1136 and a NAT gateway 1138.

[0272] The control plane VCN 1116 may include a data plane mirror application layer 1140 that may include one or more application subnets 1126. The one or more application subnets 1126 included in the data plane mirror application layer 1140 may include virtual network interface controllers (VNICs) 1142 that may execute compute instances 1144. The compute instances 1144 may communicatively couple the one or more application subnets 1126 of the data plane mirror application layer 1140 to the one or more application subnets 1126 that may be included in the data plane application layer 1146.

[0273] The data plane VCN 1118 may include a data plane application layer 1146, a data plane DMZ layer 1148, and a data plane data layer 1150. The data plane DMZ layer 1148 may include one or more LB subnets 1122 that may be communicatively coupled to the one or more application subnets 1126 of the data plane application layer 1146 and an Internet gateway 1134 of the data plane VCN 1118. The one or more application subnets 1126 may be communicatively coupled to a service gateway 1136 of the data plane VCN 1118 and a NAT gateway 1138 of the data plane VCN 1118. The data plane data layer 1150 may further include one or more DB subnets 1130 that may be communicatively coupled to the one or more application subnets 1126 of the data plane application layer 1146.

[0274] The Internet gateway 1134 for the control plane VCN 1116 and the data plane VCN 1118 can be communicatively coupled to the metadata management service 1152, which can be communicatively coupled to the public Internet 1154. The public Internet 1154 can be communicatively coupled to the NAT gateway 1138 for the control plane VCN 1116 and the data plane VCN 1118. The service gateway 1136 for the control plane VCN 1116 and the data plane VCN 1118 can be communicatively coupled to the cloud service 1156.

[0275] In some examples, the service gateway 1136 for the control plane VCN 1116 or the data plane VCN 1118 can make application programming interface (API) calls to the cloud service 1156 without going through the public Internet 1154. The API calls from the service gateway 1136 to the cloud service 1156 can be one-way: the service gateway 1136 can make API calls to the cloud service 1156, and the cloud service 1156 can send the requested data to the service gateway 1136. However, the cloud service 1156 may not initiate API calls to the service gateway 1136.

[0276] In some examples, the secure host lease 1104 can be directly connected to the service lease 1119, which would otherwise be isolated. The secure host subnet 1108 can communicate with the SSH subnet 1114 via the LPG 1110, and the LPG 1110 can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1108 to the SSH subnet 1114 can enable the secure host subnet 1108 to access other entities within the service lease 1119.

[0277] The control plane VCN 1116 can allow users of the service lease 1119 to set or otherwise provision the desired resources. The desired resources provisioned in the control plane VCN 1116 can be deployed or otherwise used in the data plane VCN 1118. In some examples, the control plane VCN 1116 can be isolated from the data plane VCN 1118, and the data plane mirror application layer 1140 of the control plane VCN 1116 can communicate with the data plane application layer 1146 of the data plane VCN 1118 via the VNIC 1142, which can be included in both the data plane mirror application layer 1140 and the data plane application layer 1146.

[0278] In some examples, a user or customer of the system can make requests, such as create, read, update, or delete (CRUD) operations, via a public Internet 1154 that can pass the requests to a metadata management service 1152. The metadata management service 1152 can pass the requests to a control plane VCN 1116 via an Internet gateway 1134. The requests can be received by one or more LB subnets 1122 included in a control plane DMZ layer 1120. The one or more LB subnets 1122 can determine that the requests are valid, and in response to that determination, the one or more LB subnets 1122 can pass the requests to one or more application subnets 1126 included in a control plane application layer 1124. If the requests are verified and a call to the public Internet 1154 is required, then the call to the public Internet 1154 can be passed to a NAT gateway 1138 that can make calls to the public Internet 1154. Metadata that the requests may expect to be stored can be stored in one or more DB subnets 1130.

[0279] In some examples, a data plane mirror application layer 1140 can facilitate direct communication between a control plane VCN 1116 and a data plane VCN 1118. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 1118. Via a VNIC 1142, the control plane VCN 1116 can communicate directly with resources included in the data plane VCN 1118, and thereby can perform configuration changes, updates, or other suitable modifications to the resources.

[0280] In some embodiments, a control plane VCN 1116 and a data plane VCN 1118 can be included in a service tenancy 1119. In such a case, a user or customer of the system may not own or operate the control plane VCN 1116 or the data plane VCN 1118. Instead, an IaaS provider can own or operate the control plane VCN 1116 and the data plane VCN 1118, both of which planes can be included in the service tenancy 1119. This embodiment can enable network isolation that can prevent a user or customer from interacting with resources of other users or other customers. Additionally, this embodiment can allow a user or customer of the system to privately store databases without relying on the public Internet 1154, which may not have a desired level of threat protection, for storage.

[0281] In other embodiments, the (one or more) LB subnets 1122 included in the control plane VCN 1116 may be configured to receive signals from the service gateway 1136. In this embodiment, the control plane VCN 1116 and the data plane VCN 1118 may be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1154. A customer of the IaaS provider may desire this embodiment because the (one or more) databases used by the customer may be controlled by the IaaS provider and may be stored on the service lease 1119, which may be isolated from the public Internet 1154.

[0282] ​ is a block diagram 1200 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1202 (e.g., ​ the service operator 1102) may be communicatively coupled to a secure host lease 1204 (e.g., ​ the secure host lease 1104), which may include a virtual cloud network (VCN) 1206 (e.g., ​ the VCN 1106) and a secure host subnet 1208 (e.g., ​ the secure host subnet 1108). The VCN 1206 may include a local peering gateway (LPG) 1210 (e.g., ​ the LPG 1110), which may be communicatively coupled to a secure shell (SSH) VCN 1212 (e.g., ​ the SSH VCN 1112) via the LPG 1110 included in the SSH VCN 1212. The SSH VCN 1212 may include an SSH subnet 1214 (e.g., ​ the SSH subnet 1114), and the SSH VCN 1212 may be communicatively coupled to a control plane VCN 1216 (e.g., ​ the control plane VCN 1116) via the LPG 1210 included in the control plane VCN 1216. The control plane VCN 1216 may be included in a service lease 1219 (e.g., ​ the service lease 1119), and a data plane VCN 1218 (e.g., ​ the data plane VCN 1118) may be included in a customer lease 1221 that may be owned or operated by a user or customer of the system.

[0283] The control plane VCN 1216 may include the (one or more) LB subnets 1222 (e.g., ​ of the control plane DMZ layer 1220 of the (one or more) LB subnets 1122 (e.g., ​ of the control plane DMZ layer 1120), may include the (one or more) application subnets 1226 (e.g., ​ of the (one or more) application subnets 1126) of the control plane application layer 1224 (e.g., ​ of the control plane application layer 1124), may include the (one or more) database (DB) subnets 1230 (e.g., similar to ​ of the (one or more) DB subnets 1130) of the control plane data layer 1228 (e.g., ​ of the control plane data layer 1128). The (one or more) LB subnets 1222 included in the control plane DMZ layer 1220 may be communicatively coupled to the (one or more) application subnets 1226 included in the control plane application layer 1224 and the Internet gateway 1234 that may be included in the control plane VCN 1216 (e.g., ​ of the Internet gateway 1134), and the (one or more) application subnets 1226 may be communicatively coupled to the (one or more) DB subnets 1230 included in the control plane data layer 1228, as well as the service gateway 1236 (e.g., ​ of the service gateway 1136) and the network address translation (NAT) gateway 1238 (e.g., ​ of the NAT gateway 1138). The control plane VCN 1216 may include the service gateway 1236 and the NAT gateway 1238.

[0284] The control plane VCN 1216 may include the data plane mirror application layer 1240 that may include the (one or more) application subnets 1226 (e.g., ​ of the data plane mirror application layer 1140). The (one or more) application subnets 1226 included in the data plane mirror application layer 1240 may include the virtual network interface controller (VNIC) 1242 that may execute the compute instance 1244 (e.g., similar to ​ of the compute instance 1144) (e.g., the VNIC of 1142). The compute instance 1244 may facilitate communication between the (one or more) application subnets 1226 of the data plane mirror application layer 1240 and the (one or more) application subnets 1226 that may be included in the data plane application layer 1246 (e.g., ​ of the data plane application layer 1146) via the VNIC 1242 included in the data plane mirror application layer 1240 and the VNIC 1242 included in the data plane application layer 1246.

[0285] The Internet gateway 1234 included in the control plane VCN 1216 can be communicatively coupled to a metadata management service 1252 (e.g., ​ the metadata management service 1152), which can be communicatively coupled to the public Internet 1254 (e.g., ​ the public Internet 1154). The public Internet 1254 can be communicatively coupled to the NAT gateway 1238 included in the control plane VCN 1216. The service gateway 1236 included in the control plane VCN 1216 can be communicatively coupled to a cloud service 1256 (e.g., ​ the cloud service 1156).

[0286] In some examples, the data plane VCN 1218 can be included in a customer tenancy 1221. In such a case, the IaaS provider can provide a control plane VCN 1216 for each customer, and the IaaS provider can provision a unique compute instance 1244 included in a service tenancy 1219 for each customer. Each compute instance 1244 can permit communication between the control plane VCN 1216 included in the service tenancy 1219 and the data plane VCN 1218 included in the customer tenancy 1221. The compute instance 1244 can permit resources provisioned in the control plane VCN 1216 included in the service tenancy 1219 to be deployed or otherwise used in the data plane VCN 1218 included in the customer tenancy 1221.

[0287] In other examples, a customer of the IaaS provider can have a database residing in the customer tenancy 1221. In this example, the control plane VCN 1216 can include a data plane mirror application layer 1240 that can include one or more application subnets 1226. The data plane mirror application layer 1240 can reside in the data plane VCN 1218, but the data plane mirror application layer 1240 can not reside in the data plane VCN 1218. That is, the data plane mirror application layer 1240 can access the customer tenancy 1221, but the data plane mirror application layer 1240 can not exist in the data plane VCN 1218 or be owned or operated by a customer of the IaaS provider. The data plane mirror application layer 1240 can be configured to make calls to the data plane VCN 1218, but can not be configured to make calls to any entity included in the control plane VCN 1216. A customer may desire to deploy or otherwise use resources provisioned in the control plane VCN 1216 in the data plane VCN 1218, and the data plane mirror application layer 1240 can facilitate the customer's desired deployment or other use of the resources.

[0288] In some embodiments, a customer of an IaaS provider may apply filters to the data plane VCN 1218. In this embodiment, the customer may determine what the data plane VCN 1218 can access, and the customer may restrict access from the data plane VCN 1218 to the public Internet 1254. The IaaS provider may not be able to apply filters or otherwise control the access of the data plane VCN 1218 to any external network or database. Applying filters and controls by the customer to the data plane VCN 1218 included in the customer lease 1221 can help isolate the data plane VCN 1218 from other customers and the public Internet 1254.

[0289] In some embodiments, a cloud service 1256 may be invoked by a service gateway 1236 to access the public Internet 1254, the control plane VCN 1216, or services that may not be present on the data plane VCN 1218. The connection between the cloud service 1256 and the control plane VCN 1216 or the data plane VCN 1218 may not be real-time or continuous. The cloud service 1256 may exist on a different network owned or operated by the IaaS provider. The cloud service 1256 may be configured to receive calls from the service gateway 1236 and may be configured not to receive calls from the public Internet 1254. Some cloud services 1256 may be isolated from other cloud services 1256, and the control plane VCN 1216 may be isolated from cloud services 1256 that may not be in the same region as the control plane VCN 1216. For example, the control plane VCN 1216 may be located in "Region 1", and the cloud service "Deployment 11" may be located in Region 1 and "Region 2". If a call is made by the service gateway 1236 included in the control plane VCN 1216 located in Region 1 to Deployment 11, then the call may be routed to Deployment 11 in Region 1. In this example, the control plane VCN 1216 or Deployment 11 in Region 1 may not be communicatively coupled or otherwise communicate with Deployment 11 in Region 2.

[0290] ​ is a block diagram 1300 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1302 (e.g., ​ service operator 1102) may be communicatively coupled to a secure host lease 1304 (e.g., ​ secure host lease 1104), which may include a virtual cloud network (VCN) 1306 (e.g., ​ VCN 1106) and a secure host subnet 1308 (e.g., ​ Secure host subnet 1108). VCN 1306 may include an LPG 1310 that can be communicatively coupled to the SSH VCN 1312 (e.g., ​ the LPG 1310 of the SSH VCN 1112) (e.g., ​ the LPG 1110). The SSH VCN 1312 may include an SSH subnet 1314 (e.g., ​ the SSH subnet 1114), and the SSH VCN 1312 may be communicatively coupled to the control plane VCN 1316 via an LPG 1310 included in the control plane VCN 1316 (e.g., ​ the control plane VCN 1116) and coupled to the data plane VCN 1318 via an LPG 1310 included in the data plane VCN 1318 (e.g., ​ the data plane 1118). The control plane VCN 1316 and the data plane VCN 1318 may be included in a service tenancy 1319 (e.g., ​ the service tenancy 1119).

[0291] The control plane VCN 1316 may include a control plane DMZ layer 1320 that may include one or more load balancer (LB) subnets 1322 (e.g., ​ one or more LB subnets 1122), a control plane application layer 1324 that may include one or more application subnets 1326 (e.g., similar to ​ one or more application subnets 1126), and a control plane data layer 1328 that may include one or more DB subnets 1330 (e.g., ​ one or more application subnets 1126) (e.g., ​ the control plane application layer 1124), a control plane data layer 1328 that may include one or more DB subnets 1330 (e.g., ​ the control plane data layer 1128). The one or more LB subnets 1322 included in the control plane DMZ layer 1320 may be communicatively coupled to the one or more application subnets 1326 included in the control plane application layer 1324 and an Internet gateway 1334 that may be included in the control plane VCN 1316 (e.g., ​ the Internet gateway 1134), and the one or more application subnets 1326 may be communicatively coupled to the one or more DB subnets 1330 included in the control plane data layer 1328, as well as a service gateway 1336 (e.g., ​ the service gateway) and a network address translation (NAT) gateway 1338 (e.g., ​ The NAT gateway 1138). The control plane VCN 1316 may include a service gateway 1336 and a NAT gateway 1338.

[0292] The data plane VCN 1318 may include a data plane application layer 1346 (e.g., ​ the data plane application layer 1146), a data plane DMZ layer 1348 (e.g., ​ the data plane DMZ layer 1148), and a data plane data layer 1350 (e.g., ​ the data plane data layer 1150). The data plane DMZ layer 1348 may include one or more trusted application subnets 1360 and one or more untrusted application subnets 1362 that may be communicatively coupled to the data plane application layer 1346 and one or more LB subnets 1322 of the Internet gateway 1334 included in the data plane VCN 1318. One or more trusted application subnets 1360 may be communicatively coupled to the service gateway 1336 included in the data plane VCN 1318, the NAT gateway 1338 included in the data plane VCN 1318, and one or more DB subnets 1330 included in the data plane data layer 1350. One or more untrusted application subnets 1362 may be communicatively coupled to the service gateway 1336 included in the data plane VCN 1318 and one or more DB subnets 1330 included in the data plane data layer 1350. The data plane data layer 1350 may include one or more DB subnets 1330 that may be communicatively coupled to the service gateway 1336 included in the data plane VCN 1318.

[0293] (One or more) untrusted application subnets 1362 may include one or more primary VNICs 1364(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1366(1)-(N). Each tenant VM 1366(1)-(N) may be communicatively coupled to a corresponding application subnet 1367(1)-(N) that may be included in a corresponding container egress VCN 1368(1)-(N), and the corresponding container egress VCN 1368(1)-(N) may be included in a corresponding customer lease 1370(1)-(N). The corresponding secondary VNICs 1372(1)-(N) may facilitate communication between one or more untrusted application subnets 1362 included in the data plane VCN 1318 and the application subnets included in the container egress VCNs 1368(1)-(N). Each container egress VCN 1368(1)-(N) may include a NAT gateway 1338, and the NAT gateway 1338 may be communicatively coupled to the public Internet 1354 (e.g., ​ the public Internet 1154).

[0294] An Internet gateway 1334 included in the control plane VCN 1316 and an Internet gateway 1334 included in the data plane VCN 1318 can be communicatively coupled to a metadata management service 1352 (e.g., ​ the metadata management system 1152), and the metadata management service can be communicatively coupled to the public Internet 1354. The public Internet 1354 can be communicatively coupled to a NAT gateway 1338 included in the control plane VCN 1316 and an NAT gateway 1338 included in the data plane VCN 1318. A service gateway 1336 included in the control plane VCN 1316 and a service gateway 1336 included in the data plane VCN 1318 can be communicatively coupled to a cloud service 1356.

[0295] In some embodiments, the data plane VCN 1318 can be integrated with a customer lease 1370. In some cases, such as when it may be desirable to support during code execution, this integration may be useful or desirable for customers of the IaaS provider. The customer can provide code that may be disruptive, may communicate with other customer resources, or may otherwise have an undesired effect to run. In response to this, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0296] In some examples, a customer of the IaaS provider can grant temporary network access to the IaaS provider and request a function to be attached to the data plane application layer 1346. The code running the function can be executed in the VMs 1366(1)-(N), and the code can not be configured to run anywhere else on the data plane VCN 1318. Each VM 1366(1)-(N) can be connected to a customer lease 1370. The corresponding containers 1371(1)-(N) included in the VMs 1366(1)-(N) can be configured to run the code. In this case, there can be double isolation (e.g., the containers 1371(1)-(N) run the code, where the containers 1371(1)-(N) may be at least included in the VMs 1366(1)-(N) included in one or more untrusted application subnets 1362), and this double isolation can help prevent incorrect or otherwise undesired code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 1371(1)-(N) can be communicatively coupled to the customer lease 1370 and can be configured to transmit or receive data from the customer lease 1370. The containers 1371(1)-(N) can not be configured to transmit or receive data from any other entity in the data plane VCN 1318. After the code execution is completed, the IaaS provider can terminate or otherwise dispose of the containers 1371(1)-(N).

[0297] In some embodiments, the (one or more) trusted application subnets 1360 may run code that may be owned or operated by an IaaS provider. In this embodiment, the (one or more) trusted application subnets 1360 may be communicatively coupled to the (one or more) DB subnets 1330 and configured to perform CRUD operations in the (one or more) DB subnets 1330. The (one or more) untrusted application subnets 1362 may be communicatively coupled to the (one or more) DB subnets 1330, but in this embodiment, the (one or more) untrusted application subnets may be configured to perform read operations in the (one or more) DB subnets 1330. Containers 1371(1)-(N) that may be included in each customer's VM 1366(1)-(N) and may run code from the customer may not be communicatively coupled to the (one or more) DB subnets 1330.

[0298] In other embodiments, the control plane VCN 1316 and the data plane VCN 1318 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1316 and the data plane VCN 1318. However, communication may occur indirectly through at least one method. An LPG 1310 may be established by the IaaS provider, and the LPG may facilitate communication between the control plane VCN 1316 and the data plane VCN 1318. In another example, the control plane VCN 1316 or the data plane VCN 1318 may make a call to a cloud service 1356 via a service gateway 1336. For example, a call from the control plane VCN 1316 to the cloud service 1356 may include a request for a service that may communicate with the data plane VCN 1318.

[0299] ​ is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1402 (e.g., ​ the service operator 1102) may be communicatively coupled to a secure host lease 1404 (e.g., ​ the secure host lease 1104), and the secure host lease 1404 may include a virtual cloud network (VCN) 1406 (e.g., ​ the VCN 1106) and a secure host subnet 1408 (e.g., ​ the secure host subnet 1108). The VCN 1406 may include an LPG 1410 (e.g., ​ the LPG 1110), and the LPG may communicate via an SSH VCN 1412 (e.g., ​ The LPG 1410 in the SSH VCN 1112 is communicatively coupled to the SSH VCN 1412. The SSH VCN 1412 may include an SSH subnet 1414 (e.g., ​ the SSH subnet 1114), and the SSH VCN 1412 may be communicatively coupled to the control plane VCN 1416 via the LPG 1410 included in the control plane VCN 1416 (e.g., ​ the control plane VCN 1116) and coupled to the data plane VCN 1418 via the LPG 1410 included in the data plane VCN 1418 (e.g., ​ the data plane 1118). The control plane VCN 1416 and the data plane VCN 1418 may be included in a service tenancy 1419 (e.g., ​ the service tenancy 1119).

[0300] The control plane VCN 1416 may include a control plane DMZ layer 1420 that may include one or more LB subnets 1422 (e.g., ​ the one or more LB subnets 1122), a control plane application layer 1424 that may include one or more application subnets 1426 (e.g., ​ the control plane DMZ layer 1120), a control plane application layer 1424 that may include one or more application subnets 1426 (e.g., ​ the one or more application subnets 1126), a control plane data layer 1428 that may include one or more DB subnets 1430 (e.g., ​ the control plane application layer 1124), a control plane data layer 1428 that may include one or more DB subnets 1430 (e.g., ​ the one or more DB subnets 1330). The one or more LB subnets 1422 included in the control plane DMZ layer 1420 may be communicatively coupled to the one or more application subnets 1426 included in the control plane application layer 1424 and the Internet gateway 1434 that may be included in the control plane VCN 1416 (e.g., ​ the control plane data layer 1128). The one or more LB subnets 1422 included in the control plane DMZ layer 1420 may be communicatively coupled to the one or more application subnets 1426 included in the control plane application layer 1424 and the Internet gateway 1434 (e.g., ​ the Internet gateway 1134), and the one or more application subnets 1426 may be communicatively coupled to the one or more DB subnets 1430 included in the control plane data layer 1428, as well as the service gateway 1436 (e.g., ​ the service gateway) and the network address translation (NAT) gateway 1438 (e.g., ​ the NAT gateway 1138). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0301] The data plane VCN 1418 may include a data plane application layer 1446 (e.g., ​ 's data plane application layer 1146), a data plane DMZ layer 1448 (e.g., ​ 's data plane DMZ layer 1148)), and a data plane data layer 1450 (e.g., ​ 's data plane data layer 1150). The data plane DMZ layer 1448 may include one or more trusted application subnets 1460 (e.g., ​ 's one or more trusted application subnets 1360) and one or more untrusted application subnets 1462 (e.g., ​ 's one or more untrusted application subnets 1362) that are communicatively coupled to the data plane application layer 1446, and one or more LB subnets 1422 of the Internet gateway 1434 included in the data plane VCN 1418. The one or more trusted application subnets 1460 may be communicatively coupled to a service gateway 1436 included in the data plane VCN 1418, a NAT gateway 1438 included in the data plane VCN 1418, and one or more DB subnets 1430 included in the data plane data layer 1450. The one or more untrusted application subnets 1462 may be communicatively coupled to a service gateway 1436 included in the data plane VCN 1418 and one or more DB subnets 1430 included in the data plane data layer 1450. The data plane data layer 1450 may include one or more DB subnets 1430 that are communicatively coupled to a service gateway 1436 included in the data plane VCN 1418.

[0302] The one or more untrusted application subnets 1462 may include primary VNICs 1464(1)-(N) that are communicatively coupled to tenant virtual machines (VMs) 1466(1)-(N) residing within the one or more untrusted application subnets 1462. Each tenant VM 1466(1)-(N) may run code in a corresponding container 1467(1)-(N) and be communicatively coupled to an application subnet 1426 in the data plane application layer 1446 that may be included in a container egress VCN 1468. Corresponding secondary VNICs 1472(1)-(N) may facilitate communication between the one or more untrusted application subnets 1462 included in the data plane VCN 1418 and the application subnet included in the container egress VCN 1468. The container egress VCN may include a NAT gateway 1438 that is communicatively coupled to a public Internet 1454 (e.g., ​ 's public Internet 1154).

[0303] The Internet gateway 1434 included in the control plane VCN 1416 and the Internet gateway 1434 included in the data plane VCN 1418 can be communicatively coupled to a metadata management service 1452 (e.g., ​ the metadata management system 1152), and the metadata management service can be communicatively coupled to the public Internet 1454. The public Internet 1454 can be communicatively coupled to the NAT gateway 1438 included in the control plane VCN 1416 and the NAT gateway 1438 included in the data plane VCN 1418. The service gateway 1436 included in the control plane VCN 1416 and the service gateway 1436 included in the data plane VCN 1418 can be communicatively coupled to the cloud service 1456.

[0304] In some examples, the pattern illustrated by the architecture of the block diagram 1400 of ​ can be considered an exception to the pattern illustrated by the architecture of the block diagram 1300 of ​ and may be desirable for customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The customer can access in real time the respective containers 1467(1)-(N) included in each customer's VMs 1466(1)-(N). The containers 1467(1)-(N) can be configured to make calls to the respective secondary VNICs 1472(1)-(N) included in the (one or more) application subnets 1426 of the data plane application layer 1446, and the data plane application layer 1446 can be included in the container egress VCN 1468. The secondary VNICs 1472(1)-(N) can forward the calls to the NAT gateway 1438, and the NAT gateway 1438 can forward the calls to the public Internet 1454. In this example, the containers 1467(1)-(N) that can be accessed by the customer in real time can be isolated from the control plane VCN 1416 and can be isolated from other entities included in the data plane VCN 1418. The containers 1467(1)-(N) can also be isolated from resources from other customers.

[0305] In other examples, a customer can use containers 1467(1)-(N) to invoke cloud service 1456. In this example, the customer can run code in containers 1467(1)-(N) that requests services from cloud service 1456. Containers 1467(1)-(N) can transmit the request to secondary VNICs 1472(1)-(N), which can transmit the request to a NAT gateway, which can transmit the request to public Internet 1454. Public Internet 1454 can transmit the request via Internet gateway 1434 to (one or more) LB subnets 1422 included in control plane VCN 1416. In response to determining that the request is valid, (one or more) LB subnets can transmit the request to (one or more) application subnets 1426, which can transmit the request to cloud service 1456 via service gateway 1436.

[0306] It should be recognized that the IaaS architectures 1100, 1200, 1300, 1400 depicted in the figures may have other components in addition to those depicted. Additionally, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that may incorporate the embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0307] In certain embodiments, the IaaS systems described herein may include application suite, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0308] ​ An example computer system 1500 in which various embodiments may be implemented is illustrated. System 1500 may be used to implement any of the computer systems described above. As shown, computer system 1500 includes a processing unit 1504 communicating with a plurality of peripheral subsystems via a bus subsystem 1502. These peripheral subsystems may include a processing acceleration unit 1506, an I / O subsystem 1508, a storage subsystem 1518, and a communication subsystem 1524. Storage subsystem 1518 includes a tangible computer-readable storage medium 1522 and system memory 1510.

[0309] The bus subsystem 1502 provides a mechanism for enabling the various components and subsystems of the computer system 1500 to communicate with each other as intended. Although the bus subsystem 1502 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1502 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard.

[0310] The processing unit 1504, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1500. One or more processors can be included in the processing unit 1504. These processors can include single-core or multi-core processors. In certain embodiments, the processing unit 1504 can be implemented as one or more independent processing units 1532 and / or 1534, each including a single-core or multi-core processor. In other embodiments, the processing unit 1504 can also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0311] In various embodiments, the processing unit 1504 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in the (one or more) processors 1504 and / or the storage subsystem 1518. Through appropriate programming, the (one or more) processors 1504 can provide the various functions described above. The computer system 1500 can additionally include a processing acceleration unit 1506, which can include a digital signal processor (DSP), a dedicated processor, and the like.

[0312] The I / O subsystem 1508 can include user interface input devices and user interface output devices. User interface input devices can include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keyboard, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can include, for example, motion sensing and / or gesture recognition devices, such as Microsoft A motion sensor that enables a user to control and interact with an input device such as a Microsoft 360 game controller through a natural user interface using gestures and voice commands. The user interface input device may also include an eye gesture recognition device, such as one that detects eye activity from the user (e.g., "blinking" when taking a photo and / or making a menu selection) and converts the eye gesture into an input to the input device (e.g., a Google Blink Detector). Additionally, the user interface input device may include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., a Navigator) through voice commands. The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad and graphics tablet, and audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. Additionally, the user interface input device may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. The user interface input device may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0313]

[0314] The user interface output device may include a display subsystem, indicator lights, or a non-visual display such as an audio output device, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one using a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1500 to the user or other computers. For example, the user interface output device may include, but is not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0315] The computer system 1500 may include a storage subsystem 1518 that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which when executed by one or more cores or processors of the processing unit 1504, provide the above functionality. The storage subsystem 1518 may also provide a repository for storing data used in accordance with this disclosure.

[0316] As ​ depicted in the example of, the storage subsystem 1518 can include various components, including system memory 1510, computer-readable storage medium 1522, and computer-readable storage medium reader 1520. The system memory 1510 can store program instructions that can be loaded and executed by the processing unit 1504. The system memory 1510 can also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs can be loaded into the system memory 1510, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0317] The system memory 1510 can also store an operating system 1516. Examples of the operating system 1516 can include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems, such as iOS, Phone, OS, OS, and OS operating systems. In certain embodiments where the computer system 1500 executes one or more virtual machines, the virtual machines along with the guest operating system (GOS) can be loaded into the system memory 1510 and executed by one or more processors or cores of the processing unit 1504.

[0318] The system memory 1510 can be configured differently depending on the type of the computer system 1500. For example, the system memory 1510 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 1510 can include a basic input / output system (BIOS), which contains basic routines that help transfer information between components within the computer system 1500 during startup.

[0319] The computer-readable storage medium 1522 can represent a remote, local, fixed, and / or removable storage device, as well as a storage medium for temporarily and / or more permanently accommodating and storing computer-readable information for use by the computer system 1500, including instructions executable by the processing unit 1504 of the computer system 1500.

[0320] The computer-readable storage medium 1522 can include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for the storage and / or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic tape cartridges, magnetic tape, disk storage devices, or other magnetic storage devices, or other tangible computer-readable media.

[0321] As an example, the computer-readable storage medium 1522 can include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk (such as a CD ROM, DVD, and disk or other optical media). The computer-readable storage medium 1522 can include, but is not limited to, drives, flash cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital audio tapes, and so on. The computer-readable storage medium 1522 can also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), SSDs based on volatile memory (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drive and its associated computer-readable medium can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1500.

[0322] Machine-readable instructions executable by one or more processors or cores of processing unit 1504 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, which include volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0323] Communication subsystem 1524 provides an interface to other computer systems and networks. Communication subsystem 1524 serves as an interface for receiving data from other systems and transmitting data from computer system 1500 to other systems. For example, communication subsystem 1524 may enable computer system 1500 to connect to one or more devices via the Internet. In some embodiments, communication subsystem 1524 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution) advanced data network technologies, WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, as an addition or alternative to the wireless interface, communication subsystem 1524 may provide wired network connectivity (e.g., Ethernet).

[0324] In some embodiments, communication subsystem 1524 may also receive input communications in the form of structured and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc., on behalf of one or more users who may use computer system 1500.

[0325] By way of example, communication subsystem 1524 may be configured to receive data feeds 1526 from users of social networks and / or other communication services in real time, such as feeds, updates, web feeds such as rich site summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0326] Additionally, the communication subsystem 1524 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1528 and / or event updates 1530 of real-time events that can be continuous or unbounded in nature and have no explicit termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and so on.

[0327] The communication subsystem 1524 may also be configured to output structured and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc., to one or more databases that may communicate with one or more streaming data source computers coupled to the computer system 1500.

[0328] The computer system 1500 may be one of various types, including handheld portable devices (e.g., cellular phones, computing tablets, PDAs), wearable devices (e.g., Glass head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0329] Due to the ever-changing nature of computers and networks, the description of the computer system 1500 depicted in the figure is merely intended as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, custom hardware may also be used and / or specific elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices such as network input / output devices may also be employed. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will recognize other ways and / or methods of implementing various embodiments.

[0330] Although specific embodiments have been described, various modifications, changes, alternative constructs, and equivalent forms are also included within the scope of the present disclosure. The embodiments are not limited to operating within certain specific data processing environments, but can operate freely within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art should appreciate that the scope of the present disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments may be used alone or in combination.

[0331] In addition, while embodiments have been described using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments can be implemented using only hardware, or only software, or using a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such a configuration can be accomplished by, for example, designing electronic circuitry to perform the operations, programming a programmable electronic circuit such as a microprocessor to perform the operations, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes can use different techniques, or the same pair of processes can use different techniques at different times.

[0332] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is clear that additions, subtractions, deletions, and other modifications and changes can be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of the following claims.

[0333] In the context of describing the disclosed embodiments (notably in the context of the following claims), the terms "a", "an", "the", and similar references are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms "comprising", "having", "including", and "containing" are to be construed as open-ended terms (i.e., meaning "including but not limited to"). The term "connected" should be construed as being at least partially incorporated in, attached to, or joined together, even if there is something in between. Unless otherwise indicated herein, the recitation of a range of values herein is merely intended to be a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated into the specification as if it were individually recited herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples, or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate the embodiments and does not pose a limitation on the scope of the present disclosure claimed, unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.

[0334] Disjunctive language, such as the phrase "at least one of X, Y, or Z," unless otherwise explicitly stated, is generally intended to be understood within the context in which it is commonly used to represent items, terms, etc., and can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0335] This disclosure describes preferred embodiments of the present disclosure, including the best mode known for practicing the present disclosure. Variations of those preferred embodiments will become apparent to those of ordinary skill in the art after reading the above description. Those of ordinary skill in the art should be able to appropriately adopt such variations and practice the present disclosure in a manner different from that specifically described herein. Accordingly, the present disclosure includes all modifications and equivalent forms of the subject matter recited in the appended claims as permitted by applicable law. In addition, unless otherwise indicated herein, the present disclosure includes any combination of the above elements in all possible variations thereof.

[0336] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in full herein.

[0337] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects disclosed above can be used singly or in combination. Additionally, embodiments can be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.< / uniqueid> < / realm>

Claims

1. A method, comprising: Monitoring network traffic received by a network virtualization device (NVD) in a cloud service provider infrastructure, the NVD implementing a set of one or more virtual network interface cards (VNICs) associated with a set of one or more computing instances in one or more overlay networks provided by the cloud service provider infrastructure, the network traffic destined for at least one computing instance from the set of one or more computing instances; Initiating, at least in part based on the monitoring, a protected mode for the NVD to protect the NVD from a potential distributed denial of service (DDoS) attack; And When the NVD is in the protected mode, causing one or more packets destined for the set of one or more computing instances to be redirected to a DDoS scrubber system rather than being sent to the NVD.

2. The method of claim 1, wherein initiating the protected mode for the NVD comprises: Determining the set of one or more VNICs implemented by the NVD, the set of one or more VNICs including a first VNIC associated with a first computing instance in the set of one or more computing instances, the first VNIC being associated with a first overlay address configured for the first computing instance, wherein the first overlay address is associated with a base address associated with the NVD; Creating a set of one or more shadow VNICs for the set of one or more VNICs; Associating the set of one or more shadow VNICs with the DDoS scrubber system; And Publishing information indicating the set of one or more shadow VNICs to the one or more overlay networks provided by the cloud service provider infrastructure.

3. The method according to claim 2, wherein causing the one or more packets destined for the one or more computing instances of the group to be redirected to the DDoS scrubber system comprises: Redirecting the one or more packets to the DDoS scrubber system due to the set of one or more shadow VNICs.

4. The method of claim 2 or claim 3, wherein: The set of one or more VNICs includes a plurality of VNICs; and The set of one or more shadow VNICs includes a single shadow VNIC.

5. The method of claim 2 or claim 3, wherein: The set of one or more VNICs includes a plurality of VNICs; and The set of one or more shadow VNICs includes a plurality of shadow VNICs, the plurality of shadow VNICs including a shadow VNIC corresponding to each VNIC in the plurality of VNICs.

6. The method of claim 2 or claim 3, wherein: The set of one or more VNICs includes a plurality of VNICs; and The set of one or more shadow VNICs includes a plurality of shadow VNICs, wherein the number of shadow VNICs in the plurality of shadow VNICs is less than the number of VNICs in the plurality of VNICs.

7. The method according to any one of claims 2-6, wherein the DDoS scrubber system includes at least one host machine configured to implement at least one shadow VNIC from the group of one or more shadow VNICs or at least one NVD configured to implement at least one shadow VNIC from the group of one or more shadow VNICs.

8. The method according to any one of claims 2-7, wherein: Creating the group of one or more shadow VNICs for the group of one or more VNICs includes: Creating a first shadow VNIC corresponding to the first VNIC associated with the first computing instance, and Associating the first overlay address with the first shadow VNIC; and associating the group of one or more shadow VNICs with the DDoS scrubber system includes: associating the first shadow VNIC with a base address associated with the DDoS scrubber system.

9. The method according to claim 8, wherein causing the one or more packets to be redirected to the DDoS scrubber system includes: For a first packet of the one or more packets, the first packet is sent to the first overlay address configured for the first computing instance; Determining that for the first overlay address, the first packet will be sent to the base address associated with the DDoS scrubber system; And Sending the first packet to the DDoS scrubber system.

10. The method according to any one of the preceding claims, further comprising: Performing at least one action on at least one of the one or more packets by the DDoS scrubber system; Wherein performing includes discarding the one or more packets, throttling the one or more packets, or forwarding the one or more packets to the NVD.

11. The method according to any one of the preceding claims, wherein when the network traffic is higher than a predetermined threshold, a potential distributed denial of service DDoS attack is determined, and the network traffic being higher than the predetermined threshold includes an average link utilization exceeding 80% for a continuous plurality of minutes, or a link utilization burst reaching 100% or higher for a continuous plurality of minutes.

12. The method according to any one of the preceding claims, further comprising: Exiting the protected mode for the NVD; And After exiting, for any packet sent to a computing instance from the group of one or more computing instances, sending the packet to the NVD instead of redirecting the packet to the DDoS scrubber system.

13. A method, comprising: Monitoring network traffic received by a first virtual network interface card VNIC associated with a first computing instance in an overlay network provided by a cloud service provider infrastructure, the network traffic being sent to the first computing instance; Initiating a protected mode for the first VNIC at least partially based on the monitoring to protect the first VNIC from a potential distributed denial of service DDoS attack; And When the first VNIC is in a protected mode, causing one or more packets destined for the first compute instance to be redirected to a DDoS scrubber system rather than being sent to a first network virtualization device (NVD) that implements the first VNIC; wherein the first VNIC is associated with a first overlay address configured for the first compute instance, and the first overlay address is associated with a base address associated with the NVD that implements the first VNIC.

14. The method of claim 13, wherein initiating the protected mode for the first VNIC comprises: creating a first shadow VNIC corresponding to the first VNIC associated with the first compute instance; associating the first overlay address with the first shadow VNIC; associating the first shadow VNIC with a base address associated with the DDoS scrubber system; and publishing information indicating the first shadow VNIC to the overlay network provided by the cloud service provider infrastructure.

15. The method of claim 14, wherein causing one or more packets to be redirected to the DDoS scrubber system comprises: for a first packet of the one or more packets, the first packet being destined for the first overlay address configured for the first compute instance; determining that for the first overlay address, the first packet is to be sent to the base address associated with the DDoS scrubber system; and sending the first packet to the DDoS scrubber system.

16. The method of claim 14, wherein the DDoS scrubber system comprises at least one host machine configured to implement the first shadow VNIC or at least one NVD configured to implement the first shadow VNIC.

17. The method of any one of claims 13 - 16, further comprising: performing at least one action on the one or more packets by the DDoS scrubber system; wherein performing comprises discarding the one or more packets, throttling the one or more packets, or forwarding the one or more packets to the NVD.

18. A method comprising: monitoring network traffic received by a plurality of network resources in one or more overlay networks provided by a cloud service provider infrastructure, the network traffic being destined for a first compute instance; initiating, at least in part based on the monitoring, a protected mode for a first network resource from the plurality of network resources to protect the first network resource from a potential distributed denial of service (DDoS) attack, the first network resource being associated with the first compute instance; and when the first network resource is in the protected mode, causing one or more packets destined for the first compute instance to be redirected to a DDoS scrubber system rather than being sent to the first network resource.

19. The method of claim 18, wherein: The first network resource is a network virtualization device (NVD), and the first network resource implements a first virtual network interface card (VNIC) associated with the first computing instance in a first overlay network among the one or more overlay networks, where the first VNIC enables the first computing instance to become part of the first overlay network, where the first VNIC is associated with a first overlay address configured for the first computing instance, and where the first overlay address is associated with a base address associated with the NVD; Initiating a protected mode for the NVD includes: Creating a first shadow VNIC corresponding to the first VNIC associated with the first computing instance, Associating the first overlay address with the first shadow VNIC, Associating the first shadow VNIC with a base address associated with the DDoS scrubber system, and Publishing information indicating the set of one or more shadow VNICs to the one or more overlay networks provided by the cloud service provider infrastructure; Causing one or more packets to be redirected to the DDoS scrubber system includes: For a first packet among the one or more packets, the first packet is sent to the first overlay address configured for the first computing instance, Determining that for the first overlay address, the first packet is to be sent to the base address associated with the DDoS scrubber system, and Sending the first packet to the DDoS scrubber system; and The DDoS scrubber system determines whether to forward the first packet to the NVD.

20. The method of claim 18, wherein the first network resource is a virtual network interface card (VNIC) associated with the first computing instance in a first overlay network among the one or more overlay networks, where the VNIC enables the first computing instance to become part of the first overlay network, where the VNIC is associated with a first overlay address configured for the first computing instance, and where the first overlay address is associated with a base address associated with a network virtualization device (NVD) that implements the VNIC; Initiating a protected mode for the VNIC includes: Creating a shadow VNIC corresponding to the VNIC associated with the first computing instance; Associating the first overlay address with the shadow VNIC; Associating the shadow VNIC with a base address associated with the DDoS scrubber system; and Publishing information indicating the set of one or more shadow VNICs to the one or more overlay networks provided by the cloud service provider infrastructure; Causing one or more packets to be redirected to the DDoS scrubber system includes: For a first packet among the one or more packets, the first packet is sent to the first overlay address configured for the first computing instance, Determining that for the first overlay address, the first packet is to be sent to the base address associated with the DDoS scrubber system, and Send the first packet to the DDoS scrubber system; and The DDoS scrubber system determines whether to forward the first packet to the NVD.

Images