Packet level data centric protection enforcement

Abstract

Techniques are described for performing data packet level data-centric protected enforcement. The techniques described herein are not limited to boundary-based security and definition and creation of difficult-to-maintain rules, but rather allow a user to create data-centric, intent-based policies that are enforced at different enforcement points within one or more networks. In some examples, a method includes receiving data packets at enforcement points (EPs) within one or more networks including a plurality of enforcement points (EPs); accessing enforcement data indicating allowed communications between the EP and one or more other EPs, where the data is generated according to a policy specifying how traffic flows through the one or more networks and a determination of possible data movement between at least two EPs of the plurality of EPs; and enforcing the flow of the data packet at the EP based on the data.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 582,767, filed September 14, 2023, entitled “Techniques for Packet Level Data Centric Protection Enforcement,” U.S. Provisional Patent Application No. 63 / 583,033, filed September 15, 2023, entitled “Data Security on the Cloud,” U.S. Provisional Patent Application No. 63 / 582,775, filed September 14, 2023, entitled “Application Developer Platform and Data Security,” U.S. Provisional Patent Application No. 63 / 612,943, filed December 20, 2023, entitled “Zero Trust Packet Routing,” U.S. Provisional Patent Application No. 63 / 553,056, filed February 13, 2024, entitled “Zero Trust Packet Routing Policy Language,” U.S. Provisional Patent Application No. 63 / 633,585, filed April 12, 2024, entitled “Zero Trust Packet Routing,” and U.S. Provisional Patent Application No. 63 / 691,390, filed September 6, 2024, entitled “Data Protection Using Data Zones and Signaling,” all of which are hereby incorporated by reference in their entirety for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of data security, and more specifically to packet level data centric protection enforcement. BACKGROUND

[0004] Cloud computing environments are large, complex systems that include many different components and related products / services. Protecting data that travels to / from these cloud computing environments, as well as data that travels within the cloud computing environments, can be challenging. Today, skilled network administrators / technicians create a large number of rules and different policies in an attempt to protect their data and networks. Even with all of these different rules and policies, a simple configuration error of a network rule / policy can expose a company’s sensitive data to the public. Adding to the challenge is that rules and policies must be designed and created for enforcing the rules / policies at each different layer of the network stack. Further, after the rules are created and deployed, a significant amount of time and money can be used to keep these rules up to date based on changing networks / needs. SUMMARY

[0005] The present disclosure generally relates to performing data packet level data-centric protection enforcement. Rather than being limited to perimeter-based security and defining and creating difficult to maintain rules, the techniques described herein allow users to create data-centric, intent-based policies that are enforced at different enforcement points within one or more networks.

[0006] In some examples, a zero trust software-defined network operates on top of existing cloud infrastructure, enabling secure and policy-driven communication between clients, services, and other resources within one or more networks. The network can implement zero trust principles designed to ensure that network interactions are authenticated, authorized, and encrypted to enhance security and access control. In some examples, the network can span multiple environments, including public clouds, private data centers, and on-premises locations, providing a flexible and secure network foundation for a variety of applications and services.

[0007] In some configurations, resources within the network(s) are associated with an “Origin ID” that uniquely identifies each resource. According to some examples, the Origin ID is a unique 32-bit unsigned integer that identifies a particular data class in a customer’s network, potentially across multiple clouds or on-premises equipment. In other examples, the Origin ID can be a simple distributed counter that increments with each new Origin ID generation. A “resource” can be an instance, service, or user principal within the network that can store or access data, such as but not limited to an object storage bucket, a compute instance, a database, a function, etc. According to some examples, a virtual cloud network (VCN) control plane (DSCP) assigns an Origin ID to each resource, including virtual network interface cards (VNICs), which can also be referred to herein as smartNICs, and gateways in the one or more networks. At least a portion of the resources within the network are enforcement points.

[0008] Rather than evaluating policies at a single location, enforcement points (“EPs”) evaluate policies at different locations along a route between a source and a destination associated with a packet being transmitted. In some configurations, a smartNIC is used as an enforcement point. Other devices (physical and / or virtual) can also be used. For example, gateways, instances, services, and applications can be EPs, as can other components / devices involved in interacting with data. In some examples, EPs are located at / near a source of data, at all / partial network hops, and at a destination (such as a service / application).

[0009] Generally speaking, when a packet is transmitted, an EP associated with a networking data plane can access enforcement data generated from ingress rules and / or egress rules associated with a policy. In this way, a packet is not transmitted from an EP until rules are evaluated by the EP and the EP determines that the transmission is authorized by the policy, as indicated by the enforcement data associated with the EP. According to some configurations, some EPs (e.g., smartNICs) are configured to perform L4 processing, and other EPs (e.g., gateways) are configured to perform L4 and / or L7 processing. In other configurations, EPs can be configured to perform processing at other network layers (e.g., LI-L7).

[0010] According to some configurations, a zero trust access (ZTA) service includes a rules engine and a distribution engine. The rules engine is configured to interpret a policy and generate “enforcement data” for rules specified in the policy that are enforced by different EPs. In some examples, the rules engine generates a graph that includes possible policy or assertion declaration results / outcomes for connections / neighbors (e.g., to other resources / EPs) to which data can potentially be passed. Generally speaking, the graph describes possible data movement between devices or services modeled as vertices by edges of the graph. In some cases, an edge represents a policy or assertion outcome. To create the graph, results / outcomes are determined for applying policy declarations in the policy for different connections between enforcement points (e.g., neighbors of enforcement points). For example, by referencing a node in the graph (e.g., an enforcement point in a network), it can be easily determined which other nodes (e.g., other enforcement points) the node is allowed to communicate with.

[0011] In some examples, each EP uses the enforcement data to determine how data should flow between itself and adjacent resources and other EPs (e.g., what communications are allowed or disallowed). In some examples, the enforcement data is specific to each EP. The graph model can be reconstructed and updated enforcement policies are distributed to enforcement points. The EPs use the enforcement data to determine what rules to apply when a packet is received at the EP.

[0012] According to some configurations, the distribution engine distributes “enforcement data” to different enforcement points (e.g., vertices in the figure), which can enforce rules via Layer 4 packet inspection that discovers relevant instructions using the origin ID of the reference. At each network hop, the EP accesses the enforcement data received from the distribution engine and generated by the rules engine to determine how to transmit the received packet. In other words, the EP that receives the packet uses the enforcement data to ensure that the packet is only forwarded along allowed paths. Additionally, unlike existing network security policies, the ZPR architecture allows for specifying lease range policies that can span across network and region boundaries.

[0013] Various embodiments are described herein to illustrate various features. These embodiments include various methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0014] At least one embodiment pertains to a computer-implemented method. Another embodiment pertains to a computing device comprising one or more processors and instructions that, when executed by the one or more processors, cause the computing device to perform any suitable combination of the method(s) disclosed herein. Yet another embodiment pertains to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing cluster, cause the computing cluster to perform any suitable combination of the method(s) disclosed herein.

[0015] The foregoing and other features and embodiments are more fully described in the following description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0017] Figure 1 is a high-level diagram illustrating a distributed environment of a virtual or overlay cloud network hosted by a cloud service provider infrastructure, in accordance with certain embodiments.

[0018] Figure 2 depicts a simplified architecture diagram of physical components in a physical network within a CSPI, in accordance with certain embodiments.

[0019] Figure 3 illustrates an example arrangement within a CSPI in which host machines are connected to multiple network virtualization devices (NVDs), in accordance with certain embodiments.

[0020] Figure 4Connectivity between a host machine and an NVD to provide I / O virtualization to support multi-tenancy is depicted in accordance with certain embodiments.

[0021] Figure 5 A simplified block diagram of a physical network provided by a CSPI is depicted in accordance with certain embodiments.

[0022] Figure 6 is a simplified block diagram of an environment illustrating enforcement of data packet level data-centric protection in accordance with certain embodiments.

[0023] Figure 7 is a simplified block diagram of an environment illustrating zero trust access (ZTA) service interacting with control plane(s) and data plane(s) in accordance with certain embodiments.

[0024] Figure 8 is a simplified diagram illustrating different components that can be included in a policy declaration / assertion.

[0025] Figure 9 An example method for performing data packet level data-centric protection is illustrated in accordance with aspects.

[0026] Figure 10 An example method for preparing resources to enforce a policy is illustrated in accordance with aspects.

[0027] Figure 11 An example method for performing data packet level data-centric protection enforcement is illustrated in accordance with aspects.

[0028] Figure 12 An example method for performing data packet level data-centric protection enforcement at a sending EP is illustrated in accordance with aspects.

[0029] Figure 13 An example method for performing data packet level data-centric protection enforcement at a receiving EP is illustrated in accordance with aspects.

[0030] Figure 14 is a block diagram illustrating an example pattern of an IaaS architecture in accordance with at least one embodiment.

[0031] Figure 15 is a block diagram illustrating another example pattern of an IaaS architecture in accordance with at least one embodiment.

[0032] Figure 16 is a block diagram illustrating another example pattern of an IaaS architecture in accordance with at least one embodiment.

[0033] Figure 17is a block diagram illustrating another example pattern of an IaaS architecture, in accordance with at least one embodiment.

[0034] Figure 18 is a block diagram illustrating an example computer system, in accordance with at least one embodiment. DETAILED DESCRIPTION

[0035] Current technology that controls data moving through a network typically relies on a perimeter-based security architecture to protect its digital assets. However, when the perimeter is breached and unprotected internal systems are accessed, devastating consequences can occur. Furthermore, cloud security controls such as firewalls, IAM handling, network access policies, network security groups, etc. each work in isolation, which can be difficult to set up and maintain, and can expose sensitive data with just one misconfiguration.

[0036] Rather than relying on perimeter-based security and defining and creating rules that are difficult to maintain, the technology described herein allows users to create data-centric, intent-based policies that are enforced at different enforcement points within one or more networks. According to some configurations, assertions / rules / declarations associated with a policy specify where data is allowed to travel within one or more networks. The terms "assertion," "rule," and "declaration" can be used interchangeably herein.

[0037] In contrast to previous technology that focuses on rules that specify which resources can access data, the assertions described herein focus on what can be done to the data itself. In other words, the assertions / policy declarations focus on how data can flow through one or more networks. As examples, an assertion / policy declaration can simply be "red data never leaves my tenancy," "blue data never reaches the internet," "blue data is not stored with red data," "green data never leaves data zone 2," etc. In some configurations, a policy can also include rules / assertions / declarations that specify how data moves between resources and / or resources in a data zone. In some examples, data policies can reference data zones, data attributes, and local conditions or circumstances. While example policy language definitions are discussed herein, other policy languages can also be used with the technology described herein.

[0038] Using the technology described herein, security rules can protect the flow of data through one or more networks by using the source of the data and the destination of the data. According to some examples, permission checks are evaluated that shift from runtime to compile time, saving time as data flows through one or more networks. As will be described in more detail below, a graph representing the intent of a security policy can be generated.

[0039] In some configurations, resources within one or more networks are associated with an“origin ID” that uniquely identifies the resource. According to some examples, the origin ID is a unique 32-bit unsigned integer that identifies a particular data class in a customer’s network, possibly across multiple clouds or on-premises equipment. In other examples, the origin ID can be a simple distributed counter that increments with each new origin ID generation. A“resource” can be an instance, service, or user principal within a network that can store or access data, such as but not limited to an object storage bucket, compute instance, database, function, etc. At least a portion of the resources within the network are enforcement points. According to some examples, a virtual cloud network (VCN) control plane (DSCP) assigns an origin ID to each resource, including virtual network interface cards (VNICs) (which can also be referred to herein as smartNICs) and gateways in one or more networks.

[0040] Rather than evaluating policies at a single location, enforcement points (“EPs”) evaluate policies at different locations along the route between a source and a target associated with a data packet being transmitted. In some configurations, a smartNIC is used as an enforcement point. Other devices (physical and / or virtual) can also be used. For example, gateways, instances, services, and applications can be EPs, as can other components / devices involved in interacting with data.

[0041] In some examples, EPs are located at / near a data source, at all / partial network hops, and at a target such as a service / application. Generally speaking, when a data packet is transmitted, an enforcement point associated with a networking data plane can access enforcement data generated from ingress and / or egress rules associated with a policy. In this way, a data packet is not transmitted from an enforcement point until rules are evaluated by the enforcement point and the enforcement point determines that the transmission is authorized by the policy. According to some configurations, some EPs (e.g., smartNICs) are configured to perform L4 processing, and other EPs (e.g., gateways) are configured to perform L4 and / or L7 processing. In other configurations, EPs can be configured to perform processing at other network layers (e.g., LI-L7).

[0042] According to some configurations, a zero trust access (ZTA) service includes a rules engine and a distribution engine. The rules engine is configured to interpret a policy to be enforced and generate“enforcement data” for rules specified in the policy that are enforced by different EPs throughout one or more networks.

[0043] In some examples, the rules engine generates a graph that includes possible policy or assertion declaration results / outcomes for the connections / neighbors to which data is potentially passed. Generally, the graph describes possible data movement between devices or services modeled as vertices through edges of the graph. In some cases, an edge represents a policy or assertion outcome. To create the graph, the results / outcomes of applying policy declarations in a policy for different connections between enforcement points (e.g., neighbors of an enforcement point) are determined. For example, by referencing a node in the graph (e.g., an enforcement point in a network), it can be easily determined which other nodes (e.g., other enforcement points) the node is allowed to communicate with.

[0044] In some configurations, each EP uses enforcement data to determine how data should flow between itself and adjacent resources and other EPs. Enforcement data is essentially pre-defined instructions for each EP that specify how the EP should handle data requests from any adjacent node based on a pre-evaluated set of policies associated with its origin ID. Similarly, enforcement data specifies how to handle transmission of outgoing data packets to any adjacent enforcement points or service endpoints defined in the graph based on pre-evaluated assertions associated with its origin ID. In some examples, enforcement data is specific to each EP.

[0045] In some examples, each enforcement point within a network is provided with “enforcement data” that indicates other enforcement points that the EP can communicate with according to policies. According to some configurations, a distribution engine distributes enforcement data to different enforcement points (e.g., vertices in a graph) that can use referenced origin IDs to discover relevant instructions to enforce rules. At each network hop, an EP accesses enforcement data received from the distribution engine to determine how to transmit a received data packet. In other words, an EP that receives a data packet uses enforcement data to ensure that the data packet is only forwarded along allowed paths.

[0046] When security policies change (e.g., an intent of a data administrator changes), the graph can be rebuilt and updated enforcement data sent to EPs (e.g., VNICs / smartNICs in one or more networks). Using these techniques, EPs can quickly and easily determine where data can flow, eliminating the need to make computationally more expensive determinations.

[0047] According to some configurations, zones specifying how data should be allowed to propagate can be created automatically and / or manually. A zone, which can also be referred to herein as a “data zone,” represents a collection of resources whose properties belong to a set of data zone property definitions. A data zone is a collection of resources with similar data classification properties. In some examples, a zone defaults to preventing data from being transferred outside of the zone. In other examples, a user can specify how data is allowed to flow between zones, and / or to other enforcement points.

[0048] In some examples, one or more assertions can be configured for each zone. An assertion defined for a particular data zone is a set of one or more rules that apply to a set of resources classified into or belonging to the particular data zone and that must not be violated. An assertion is an expression that governs access to data within a zone. In certain embodiments, in response to a request to transfer data from a first entity in one zone to a second entity in a second zone, any assertions configured for the zones are identified and evaluated. The evaluation of the assertions results in an allow / deny directive. The requested communication of the data is allowed only if no assertions for the zones are violated. Some example assertions that can be configured for a data zone include, but are not limited to, not allowing certain types of data to leave the zone, not sending data from the zone to a computer connected to the Internet, not allowing communication with hosts that are more than x hops away, etc.

[0049] According to some configurations, one or more data policies can be defined between two zones. A data policy between two data zones is a rule for data movement between resources classified into both of the data zones. A data policy can reference data zones, data properties, and local circumstances or conditions. In certain examples, in response to a request to transfer data from an entity in one zone to an entity in a second zone, any policies between the two zones are evaluated. The evaluation of the policies results in an allow / deny directive. The requested communication of the data is allowed if no policies between the two zones are violated. In other words, data is allowed to move between zones only if no assertions for the source zone are violated, and furthermore, no policies between the two zones through which the data is moving are violated.

[0050] In some examples, a zero trust software-defined network operates on top of existing cloud infrastructure, enabling secure and policy-driven communication between clients, services, and other resources. The architecture implements zero trust principles, designed to ensure that network interactions are authenticated, authorized, and encrypted to enhance security and access control. In some examples, the network can span multiple environments, including public clouds, private data centers, and on-premises locations, providing a flexible and secure network foundation for a variety of applications and services.

[0051] Unlike traditional role-based access control, which makes calls to trusted third-party systems to grant access at the time of the request, the technology described herein provides the EP with decisions to make ahead of the request, thereby speeding up the authorization process. Additionally, using prior art, the industry uses content inspection to monitor and filter data leaving the organization to prevent the unauthorized transmission of sensitive information. Furthermore, prior art in the industry uses policy enforcement to implement policies that specify how sensitive data should be handled and protected. However, the technology described herein enhances these processes because all possible outcomes related to the enforcement of the relevant policies between any two nodes in our organization can be pre-evaluated using a graph model of the relationships, where all possible outcomes related to the data loss prevention decision process are provided to the EP as enforcement data ahead of time to make decisions.

[0052] The data-centric security model described herein can provide significant advantages by focusing on protecting the sensitive data of an organization, regardless of where the data is stored or how the data is accessed. The data-centric security model aims to protect sensitive data, such as customer information, intellectual property, financial records, etc. This protection helps prevent data breaches and data loss, thereby reducing the risk of financial loss and damage to a company’s reputation. By focusing on data rather than just the perimeter, cloud customers can better identify vulnerabilities and threats. This enables proactive risk management and mitigation strategies, thereby reducing the likelihood and impact of security incidents. The technology described herein allows for better data protection than prior art because the technology seeks to inspect every data packet containing data and evaluate the data movement against specified data security intents.

[0053] The technology described herein also provides better visibility into how data is used and accessed throughout an organization. This insight can help detect abnormal or unauthorized data access patterns, thereby potentially identifying security vulnerabilities or insider threats. The generation of the graph also provides a graphical model, which can be particularly helpful in identifying vulnerabilities because it provides a comprehensive graphical view of a user’s assets. Labeling each data resource with a unique provenance ID also makes it easier to track and analyze data as it flows through one or more networks. The technology described herein allows for the regulation of data movement between physical and virtual devices and services based on the evaluation of policies and assertions at the data packet level.

[0054] The examples described herein provide further advantages over the prior art. Example advantages include, but are not limited to: the ability to translate a data administrator's security policy intent into enforcement data that can be interpreted by devices that act on TCP / IP packets at layer 4 or layer 7; the creation of enforcement data that is pushed to network devices with the intent of enforcing security related policies and assertions decisions at layer 4 or by services; the transmission of enforcement data to specific network devices for inline enforcement of the data administrator's intent; the use of efficient data structures (e.g., Trie data structures) to reduce the space used to store enforcement data on network devices. The use of enforcement data provided to network devices facilitates the control of data sharing with authorized parties / devices. The use of origin IDs provides a mechanism to track data access and perform anomaly detection at the packet level. The implementation of data auditing and monitoring is made more efficient by the use of enforcement data and origin IDs by network devices when inspecting TCP / IP packets at layer 4. Runtime efficiency is also improved by pre-computing all possible outcomes from a requester ahead of time, thereby eliminating the need to query in response to receiving an incoming request.

[0055] Example virtual networking architecture

[0056] The term cloud service is generally used to refer to a service provided by a cloud service provider (CSP) to a user or customer on-demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the infrastructure of the CSP are separate from the customer's own on-premise servers and systems. Thus, a customer can take advantage of the cloud services provided by the CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide simple, scalable access to applications and computing resources for subscribing customers without the customer having to invest in purchasing the infrastructure used to provide the services.

[0057] There are several cloud service providers that provide various types of cloud services. There are various different types or models of cloud services, including software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), etc.

[0058] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service provided by a CSP, a tenancy or account is created for that customer. The customer can then access the subscribed one or more cloud resources associated with that account via this account.

[0059] As noted above, Infrastructure as a Service (IaaS) is a particular type of cloud computing service. In an IaaS model, a CSP provides infrastructure (referred to as cloud service provider infrastructure or CSPI) that can be used by customers to build their own customizable networks and deploy customer resources. Thus, the customer's resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing, in which the customer's resources and networks are hosted by infrastructure provided by the customer.

[0060] The CSPI can include high performance computing resources interconnected to form a physical network, including various host machines, memory resources, and network resources, which is also referred to as a substrate network or underlay network. The resources in the CSPI can be spread across one or more data centers, which can be geographically spread across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also referred to as a software-based network, a software-defined network, or a virtual network) on top of the physical network. The CSPI physical network provides the underlay foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or substrate network or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines, etc. The overlay network is a logical (or virtual) network that runs on top of the physical substrate network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to differentiate traffic belonging to different overlay networks. Virtual or overlay networks are also referred to as virtual cloud networks (VCNs). Virtual networks are implemented using software virtualization techniques (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can run on top of the physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically either layer 3 IP networks or layer 2 VLANs. This approach to virtual or overlay networking is often referred to as virtual or overlay layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or generic routing encapsulation (GRE)), virtual extensible local area network (VXLAN - IETF RFC 7348), virtual private networks (VPNs) (e.g., MPLS layer 3 virtual private networks (RFC 4364)), VMware's NSX, GENEVE (generic network virtualization encapsulation), etc.

[0061] For IaaS, the infrastructure provided by the CSP (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider can also provision various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.). Thus, since these services can be policy driven, an IaaS user can be able to implement policies to drive load balancing to maintain application availability and performance. The CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. The CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in flexible virtual networks that are securely accessible from a variety of networked locations, such as from a customer’s on-premise network. When a customer subscribes or registers for IaaS services provided by the CSP, a tenancy created for that customer is a secure and isolated partition within the CSPI in which the customer can create, organize, and manage their cloud resources.

[0062] A customer can build their own virtual networks using the computing, memory, and networking resources provided by the CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use the resources provided by the CSPI to build one or more customizable and private virtual networks, referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as compute instances, on the customer’s VCNs. The compute instances can take the form of virtual machines, bare-metal instances, etc. Thus, the CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, virtual, hosted environment. The customer does not manage or control the underlying physical resources provided by the CSPI, but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., firewalls).

[0063] A CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In certain implementations, the console is a web-based application provided by the CSP.

[0064] CSPI can support a single-tenancy or multi-tenancy architecture. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy scenario, precautions are taken and safeguards are implemented in CSPI to ensure that each tenant’s data is isolated and remains invisible to other tenants.

[0065] In a physical network, a network endpoint (“endpoint”) refers to a computing device or system that is connected to the physical network and communicates back and forth with the network to which it is connected. Network endpoints in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by physical host machines). These endpoints in a virtual network are addressed by overlay addresses, such as overlay layer 2 addresses (e.g., overlay MAC addresses) and overlay layer 3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network managers to move around overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for the virtual network). Accordingly, unlike a physical network, in a virtual network, an overlay address (e.g., an overlay IP address) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communications between components in a virtual network involve both the virtual network and the underlying physical network. To facilitate such communications, components of CSPI are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the substrate network, and vice versa. These mappings are then used to facilitate communications. Customer traffic is encapsulated to facilitate routing in the virtual network.

[0066] Accordingly, physical addresses (e.g., physical IP addresses) are associated with components in the substrate or physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in the virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in the substrate or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in the overlay network, such as an overlay address associated with a compute instance in a customer’s virtual cloud network (VCN). Two different customers or tenants (each with their own private VCN) can potentially use the same overlay IP address in their VCNs without needing to know about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These addresses are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A VIP is used by a load balancer to map or represent multiple servers, each with its own real IP address, for example.

[0067] The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI can include components in the physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) located in virtual networks built on top of the physical network components. In certain embodiments, the CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a local geographic area containing one or more data centers. Regions are generally independent of each other and can be far apart, e.g., across countries or even continents. For example, a first region can be in Australia, another in Japan, another in India, etc. The CSPI resources are divided among the regions so that each region has its own independent subset of CSPI resources. Each region can provide a collection of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure, etc.); storage resources (e.g., block volume storage, file storage, object storage, archival storage); networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connections to on-premise networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region generally has multiple paths connecting it to other regions in the domain.

[0068] Generally, applications are deployed in the region where they are used most frequently (i.e., on the infrastructure associated with that region) because it is faster to use resources nearby than to use resources far away. Applications can also be deployed in different regions for various reasons, such as redundancy to mitigate the risk of region-wide events (such as large weather systems or earthquakes), to meet different requirements of legal jurisdictions, tax jurisdictions, and other business or social standards, etc.

[0069] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are either region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0070] ADs within a region are isolated from each other, have fault tolerance capabilities, and are configured such that they are extremely unlikely to fail at the same time. This is achieved by ADs not sharing critical infrastructure resources (such as networking, physical cables, cable paths, cable entry points, etc.) so that a failure at one AD within a region is unlikely to affect the availability of other ADs within the same region. ADs within the same region can be connected to each other through a low-latency, high-bandwidth network, which makes it possible to provide high-availability connectivity for other networks (e.g., the Internet, a customer’s on-premise network, etc.) and build replicated systems across multiple ADs to achieve high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by an IaaS provider grows, more regions and ADs and additional capacity can be added. Traffic between availability domains is typically encrypted.

[0071] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A customer’s tenancy or account with a CSP exists in a single realm and can be spread across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account is created for the customer in a region specified by the customer (called the “home” region) within a realm. The customer can extend the customer’s tenancy to one or more other regions within the realm. The customer cannot access regions that are not in the realm of the customer’s tenancy.

[0072] IaaS providers can offer multiple tenancies, each catering to a particular set of customers or users. For example, a business tenancy can be offered for business customers. As another example, a tenancy can be offered for customers in a particular country. As yet another example, a government tenancy can be offered for governments and the like. For example, a government tenancy can cater to a particular government and can have a higher level of security than a business tenancy. For example, Oracle Cloud Infrastructure (OCI) currently offers a tenancy for business regions and two tenancies for government cloud regions (e.g., FedRAMP authorized and IL5 authorized).

[0073] In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow distribution of compute instances such that the instances are not located on the same physical hardware within a single AD. This is referred to as anti-affinity. A fault domain refers to a collection of hardware components (computers, switches, etc.) that share a single point of failure. Compute pools are logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. The number of fault domains for each AD can vary depending on the embodiment. For example, in certain embodiments, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0074] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer’s tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted by CSPI in the cloud is referred to as a virtual cloud network (VCN). A customer can set up one or more virtual cloud networks (VCNs) using CSPI resources allocated to the customer. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer’s VCN can include compute instances (e.g., virtual machines, bare-metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, and the like. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) over a public network, such as the Internet, with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), with the customer’s on-premise data center or network, and with service endpoints and other types of endpoints.

[0075] CSPs can use CSPI to provide various services. In some cases, customers of CSPI can themselves act like service providers and provide services using CSPI resources. Service providers can expose service endpoints that feature identifying information (e.g., IP addresses, DNS names, and ports). A customer's resources (e.g., compute instances) can use a particular service by accessing a service endpoint exposed by the service for that particular service. These service endpoints are generally endpoints that users can publicly access via a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes also referred to as public endpoints.

[0076] In certain embodiments, a service provider can expose a service via an endpoint for the service, sometimes referred to as a service endpoint. Customers of the service can then use this service endpoint to access the service. In certain implementations, a service endpoint provided for a service can be accessed by multiple customers that intend to consume the service. In other implementations, a dedicated service endpoint can be provided for a customer such that only that customer can use the dedicated service endpoint to access the service.

[0077] In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN resides within a single region, but can span one or more or all of the availability domains in that region. Gateways are virtual interfaces configured for a VCN and enable communication of traffic between the VCN and one or more endpoints outside the VCN. One or more different types of gateways can be configured for a VCN to enable communication to and from different types of endpoints.

[0078] A VCN can be subdivided into one or more subnetworks, such as one or more subnets. Thus, a subnet is a unit or subdivision of configuration that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in that VCN and represent a subset of the address space within the VCN's address space.

[0079] Each compute instance is associated with a virtual network interface card (VNIC), which enables the compute instance to participate in a subnet of the VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC exists in a subnet, has one or more associated IP addresses, and associated security rules or policies. A VNIC is equivalent to a layer 2 port on a switch. A VNIC is attached to a compute instance and a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in different subnets in the VCN, or with endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with the compute instance. For a subnet that includes a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, each VNIC attached to a compute instance within the set of compute instances.

[0080] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay network IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs in a given subnet use the same routing table, security list, and DHCP options. As noted above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of the address space within the address space of the VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses allocated for the subnet.

[0081] In certain embodiments, in addition to a private overlay IP address, a compute instance can optionally be assigned additional overlay IP addresses, such as one or more public IP addresses if in a public subnet, for example. The multiple addresses are assigned either on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance launch and is associated with the overlay private IP address assigned to the instance - this primary VNIC cannot be deleted. Additional VNICs, referred to as secondary VNICs, can be added to existing instances in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The secondary VNICs can be in the same subnet in the same VCN as the primary VNIC, or in different subnets in the same or different VCNs.

[0082] If a compute instance is in a public subnet, it can optionally be assigned a public IP address. At the time a subnet is created, the subnet can be designated as either a public subnet or a private subnet. A private subnet means that resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs in the subnet can have public IP addresses. A customer can specify that a subnet exists in a single availability domain or across multiple availability domains in a region or realm.

[0083] As noted above, a VCN can be subdivided into one or more subnets. In certain embodiments, a virtual router (VR) configured for a VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For a subnet within a VCN, the VR represents a logical gateway for that subnet that enables the subnet (i.e., compute instances on the subnet) to communicate with endpoints on other subnets within the VCN as well as other endpoints outside the VCN. The VCN VR is a logical entity that is configured to route traffic between VNICs in the VCN and virtual gateways ("gateways") associated with the VCN. More on this below with respect to Figure 1A gateway is further described. A VCN VR is a layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, where the VCN VR has a potentially unlimited number of ports addressed by IP addresses, one for each subnet in the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address in the overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN has two subnets with associated address ranges of 10.0 / 16 and 10.1 / 16, respectively. For the first subnet in the VCN with address range 10.0 / 16, addresses in this range are reserved for the ports of the VCN VR for that subnet. In some cases, the first IP address in the range can be reserved for the VCN VR. For example, for a subnet with an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 can be reserved for the port of the VCN VR for that subnet. For the second subnet in the same VCN with address range 10.1 / 16, the VCN VR can have a port for the second subnet at IP address 10.1.0.1. The VCN VR has a different IP address for each subnet in the VCN.

[0084] In some other embodiments, each subnet within a VCN can have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. For example, the reserved or default IP address can be the first IP address in the IP address range associated with the subnet. VNICs in the subnet can communicate (e.g., send and receive packets) with the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the ingress / egress point for the subnet. The VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with gateways associated with the VCN. The VR functionality for a subnet runs on or is performed by one or more NVDs performing VNIC functionality for VNICs in the subnet.

[0085] A route table, security rules, and DHCP options can be configured for a VCN. A route table is a virtual route table for the VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN through gateways or specially configured instances. The route table for a VCN can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to instances at instance launch time.

[0086] Security rules configured for a VCN represent overlay firewall rules for the VCN. Security rules can include ingress and egress rules and specify the types of traffic (e.g., based on protocol and port) that are allowed to enter or exit a VCN instance. A customer can choose whether a given rule is stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by setting a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a collection of security rules that only apply to resources in that group. On the other hand, a security list includes rules that apply to all resources in any subnet that uses that security list. A default security list with default security rules can be provided for a VCN. DHCP options configured for a VCN provide configuration information that is automatically provided to instances in the VCN at instance launch time.

[0087] In certain embodiments, configuration information for a VCN is determined and stored by a VCN control plane. For example, configuration information for a VCN can include information about address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs in the VCN, NVDs (e.g., VNICs, VRs, gateways) that perform various virtualized network functions associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, a VCN distribution service publishes to NVDs configuration information stored by the VCN control plane or portions thereof. The distributed information can be used to update information (e.g., forwarding tables, routing tables, etc.) stored and used by the NVDs to forward packets to or from compute instances in the VCN.

[0088] In certain embodiments, creation of VCNs and subnets is handled by a VCN control plane (CP) and launch of compute instances is handled by a compute control plane. The compute control plane is responsible for allocating physical resources for compute instances and then invoking the VCN control plane to create VNICs and attach them to the compute instances. The VCN CP also sends VCN data mappings to a VCN data plane that is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of a VCN control plane are also depicted in Figure 14 、 Figure 15 、 Figure 16 and Figure 17 (see reference numbers 1416, 1516, 1616, and 1716) and described below.

[0089] A customer can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside of CSPI.

[0090] Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 Various different architectures for implementing cloud-based services using CSPI are depicted in Figure 1 is a high-level diagram of a distributed environment 100 illustrating overlay or customer VCNs hosted by CSPI, in accordance with certain embodiments. Figure 1 The distributed environment depicted in Figure 1 The distributed environment 100 depicted in Figure 1 The distributed environment depicted in Figure 1 may have more or fewer systems or components than those shown in

[0091] As shown in the example depicted in Figure 1 The distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their virtual cloud networks (VCNs), as shown in the example depicted in Figure 1 One example region, “Region US” 102, is shown in

[0092] In the embodiment depicted in Figure 1 The customer VCN 104 includes two subnets, “Subnet-1” and “Subnet-2,” each with its own CIDR IP address range, in the embodiment depicted in Figure 1In this example, the overlay IP address range for subnet-1 is 10.0 / 16, and the address range for subnet-2 is 10.1 / 16. The VCN virtual router 105 represents a logical gateway for the VCN, which enables communication between the subnets of the VCN 104 and other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between VNICs in the VCN 104 and gateways associated with the VCN 104. The VCN VR 105 provides a port for each subnet of the VCN 104. For example, the VR 105 can provide a port for subnet-1 with IP address 10.0.0.1 and a port for subnet-2 with IP address 10.1.0.1.

[0093] Multiple compute instances, which can be virtual machine instances and / or bare metal instances, can be deployed on each subnet. The compute instances in a subnet can be hosted by one or more host machines within the CSPI 101. The compute instances participate in the subnet via VNICs associated with the compute instances. For example, as shown in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances, which can be virtual machine instances or bare metal instances, can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances, which can be virtual machine instances or bare metal instances, can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in

[0094] Multiple compute instances, which can be virtual machine instances and / or bare metal instances, can be deployed on each subnet. The compute instances in a subnet can be hosted by one or more host machines within the CSPI 101. The compute instances participate in the subnet via VNICs associated with the compute instances. For example, as shown in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances, which can be virtual machine instances or bare metal instances, can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in Figure 1 In the example shown in Figure 1 In the example shown in Figure 1 In the example shown in

[0095] VCN A 104 can also include one or more load balancers. For example, a load balancer can be provisioned for a subnet and can be configured to load balance traffic across multiple compute instances on the subnet. Load balancers can also be provisioned to load balance traffic across subnets in a VCN.

[0096] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet-1); endpoints on a different subnet but within the same VCN (e.g., communication between a compute instance in Subnet-1 and a compute instance in Subnet-2); endpoints in different VCNs in the same region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet-1 and an endpoint in a service endpoint 110 in the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in subnets hosted by CSPI 101 can also communicate with endpoints that are not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in a customer’s on-premise network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 114 that are accessible via a public network such as the Internet, and other endpoints.

[0097] Communication between compute instances on the same subnet is facilitated using the VNICs associated with the source and destination compute instances. For example, compute instance Cl in subnet- 1 can want to send a packet to compute instance C2 in subnet- 1. For a packet originating from a source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance can include determining the destination information for the packet from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop with the goal of facilitating communication of the packet to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance is then executed and forwards the packet to the destination compute instance.

[0098] For packets that are to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, communication is facilitated through the VNICs associated with the source and destination compute instances and the VCN VR. For example, if compute instance Cl in subnet- 1 wants to send a packet to compute instance Dl in subnet-2, then the packet is first processed by the VNIC associated with compute instance Cl. The VNIC associated with compute instance Cl is configured to route the packet to the VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. The VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The VNIC associated with Dl then receives and processes the packet and the VNIC forwards the packet to compute instance Dl. Figure 1

[0099] ​For a data packet that is to be communicated from a compute instance in VCN 104 to an endpoint outside of VCN 104, the communication is facilitated by the VNIC associated with the source compute instance, the VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint that is external to the VCN. A gateway is a Layer 3 / IP layer concept and enables a VCN to communicate with endpoints outside of the VCN. Thus, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communications with different types of endpoints. Depending on the gateway, the communication can be over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0100] For example, compute instance CI can want to communicate with an endpoint outside of VCN 104. The data packet can first be processed by the VNIC associated with the source compute instance CI. The VNIC processing determines that the destination of the data packet is outside of CI's subnet- 1. The VNIC associated with CI can forward the data packet to the VCN VR 105 for VCN 104. The VCN VR 105 then processes the data packet and, as part of the processing, determines, based on the destination of the data packet, a particular gateway associated with VCN 104 as the next hop for the data packet. The VCN VR 105 can then forward the data packet to the particular identified gateway. For example, if the destination is an endpoint within a customer's on-premise network, then the data packet can be forwarded by the VCN VR 105 to a dynamic routing gateway (DRG) gateway 122 configured for VCN 104. The data packet can then be forwarded from the gateway to the next hop to facilitate the communication of the data packet to its ultimate intended destination.

[0101] Various different types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are depicted in Figure 1 and described below. Examples of gateways associated with a VCN are also depicted in Figure 14 , Figure 15 , Figure 16 and Figure 17 and described below. As Figure 1As shown in the depicted embodiment, a dynamic routing gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provide a path for private network traffic communication between the customer VCN 104 and another endpoint, which can be the customer's on-premise network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premise network 116 can be a customer network built using the customer's resources or a customer data center. Access to the customer on-premise network 116 is generally very restricted. For customers who have both a customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer can want their on-premise network 116 and their cloud-based VCNs 104 to be able to communicate with each other. This enables the customer to build an extended hybrid environment that includes the customer's VCNs 104 hosted by CSPI 101 and their on-premise network 116. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is set up, with one endpoint of the channel in the customer on-premise network 116 and the other endpoint in CSPI 101 and connected to the customer VCN 104. The communication channel 124 can be over a public communication network, such as the Internet, or a private communication network. Various different communication protocols can be used, such as IPsec VPN technology over a public communication network such as the Internet, Oracle's FastConnect technology that uses a private network instead of a public network, etc. The device or equipment in the customer on-premise network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as Figure 1 As shown in the depicted CPE 126. On the CSPI 101 side, the endpoint can be a host machine that executes the DRG 122.

[0102] In certain embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer VCN 104 can connect with a VCN 108 in another region using the DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft Azure cloud, the Amazon AWS cloud, etc.

[0103] As Figure 1As shown in the middle, an Internet Gateway (IGW) 120 can be configured for the customer VCN 104, which enables compute instances on the VCN 104 to communicate with public endpoints 114 that are accessible through a public network, such as the Internet. An IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. The IGW 120 enables public subnets within a VCN (such as VCN 104), where resources in the public subnets have public overlay IP addresses, to directly access public endpoints 112 on the public network 114, such as the Internet. Using the IGW 120, connections can be initiated from subnets within the VCN 104 or from the Internet.

[0104] A Network Address Translation (NAT) gateway 128 can be configured for the customer's VCN 104 and enables cloud resources in the customer's VCN that do not have a dedicated public overlay IP address to access the Internet, and does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables private subnets within a VCN (such as Private Subnet-1 in VCN 104) to privately access public endpoints on the Internet. In a NAT gateway, connections to the public Internet can only be initiated from the private subnets, not from the Internet.

[0105] In certain embodiments, a Service Gateway (SGW) 126 can be configured for the customer VCN 104 and provides a path for private network traffic between the VCN 104 and service endpoints supported in the service network 110. In certain embodiments, the service network 110 can be provided by the CSP and can provide various services. An example of such a service network is Oracle's service network, which provides various services available for customers to use. For example, a compute instance (e.g., a database system) in a private subnet of the customer VCN 104 can back up data to a service endpoint (e.g., Object Storage) without needing a public IP address or accessing the Internet. In certain embodiments, a VCN can have only one SGW, and connections can only be initiated from subnets within the VCN, not from the service network 110. If the VCN is peered with another, resources in the other VCN typically cannot access the SGW. Resources in an on-premise network connected to the VCN with FastConnect or VPN Connect can also use the service gateway configured for the VCN.

[0106] In certain embodiments, the SGW 126 uses the concept of a service Classless Inter Domain Routing (CIDR) tag, which is a string that represents all the regional public IP address ranges for a service or group of services of interest. Customers use the service CIDR tag when they configure the SGW and related routing rules to control traffic to the service. If the public IP addresses of the service change in the future, then customers can optionally use it when configuring security rules without needing to adjust them.

[0107] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 and enables the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, and traffic does not need to traverse a public network (such as the Internet) or route through the customer's on-premise network 116. In preferred embodiments, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0108] Service providers, such as the providers of services in the service network 110, can provide access to services using different access models. According to a public access model, a service can be exposed as a public endpoint that is publicly accessible by compute instances in a customer's VCN via a public network, such as the Internet, and / or can be accessed privately via the SGW 126. According to a specific private access model, a service is accessible as a private IP endpoint in a private subnet in a customer's VCN. This is referred to as private endpoint (PE) access and enables a service provider to expose its services as instances in a customer's private network. A private endpoint resource represents a service within a customer's VCN. Each PE appears as a VNIC (called a PE-VNIC, with one or more private IPs) in a subnet of the customer's VCN, selected by the customer. Thus, a PE provides a way to present a service in a private customer VCN subnet using a VNIC. Since the endpoint is exposed as a VNIC, all the features associated with a VNIC (such as routing rules, security lists, etc.) are now available for the PE VNIC.

[0109] Service providers can register their services to enable access through a PE. A provider can associate policies with a service that limit the visibility of the service to customer tenancies. A provider can register multiple services under a single virtual IP address (VIP), especially for multi-tenancy services. There can be multiple such private endpoints (in multiple VCNs) that represent the same service.

[0110] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in the service network 110) that acts as an ingress / egress point for all traffic from / to the customer subnet private endpoints. The PAGW 130 enables a provider to scale the number of PE connections without utilizing its internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer’s perspective, the PE VNIC is not attached to the customer’s instance, but appears to be attached to the service that the customer wants to interact with. Traffic to the private endpoint is routed to the service via the PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0111] By allowing traffic to flow through the FastConnect / IPsec link and the private endpoints in the customer VCN, the PE concept can also be used to extend private access for a service to a customer’s on-premise network and data center. By allowing traffic to flow between the LPG 132 and the PEs in the customer’s VCN, private access to the service can also be extended to the customer’s peered VCNs.

[0112] Customers can control routing in a VCN at the subnet level, so customers can specify which subnets in the customer’s VCN (such as the VCN 104) use each gateway. The route table of a VCN is used to decide whether to allow traffic to leave the VCN through a particular gateway. For example, in a particular instance, the route table for public subnets within the customer VCN 104 can send non-local traffic through the IGW 120. The route table for private subnets within the same customer VCN 104 can send traffic destined for CSP services through the SGW 126. All remaining traffic can be sent via the NAT gateway 128. The route table only controls traffic flowing out of the VCN.

[0113] A security list associated with a VCN is used to control traffic entering the VCN via gateways via inbound connections. All resources in a subnet use the same route table and security list. A security list can be used to control what types of traffic are allowed to and from instances in a subnet of a VCN. Security list rules can include ingress (inbound) and egress (outbound) rules. For example, an ingress rule can specify a range of allowed source addresses, while an egress rule can specify a range of allowed destination addresses. A security rule can specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In certain embodiments, an instance's operating system can enforce its own firewall rules that comply with the security list rules. The rules can be stateful (e.g., tracking connections and automatically allowing responses without an explicit security list rule for the response traffic) or stateless.

[0114] Access from a customer VCN (i.e., by resources or compute instances deployed on the VCN 104) can be classified as public access, private access, or dedicated access. Public access refers to an access model that uses public IP addresses or NAT to access public endpoints. Private access enables customer workloads in a VCN 104 (e.g., resources in a private subnet) with private IP addresses to access services without traversing a public network such as the Internet. In certain embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access services (of public service endpoints) using a service gateway. Thus, a service gateway provides a private access model by establishing a virtual link between a customer's VCN and public endpoints of services that reside outside the customer's private network.

[0115] Further, CSPI can provide dedicated public access using technologies such as FastConnect public peering, where customer on-premise instances can access one or more services in a customer VCN using a FastConnect connection without traversing a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering, where customer on-premise instances with private IP addresses can access customer VCN workloads using a FastConnect connection. FastConnect is a network connectivity alternative to using the public Internet to connect a customer's on-premise network to CSPI and its services. FastConnect provides a simple, flexible, and economical way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent network experience compared to Internet-based connections.

[0116] Figure 1The accompanying description above describes various virtualized components in an example virtual network. As described above, the virtual network is built on an underlying physical or substrate network. Figure 2 A simplified architectural diagram of physical components in a physical network within CSPI 200 providing an underlay for virtual networks is depicted in accordance with certain embodiments. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers (i.e., customers that have subscribed to one or more services provided by the CSP). Based on the services to which a customer subscribes, a subset of the resources of CSPI 200 (e.g., compute, memory, and network resources) are provisioned for the customer. The customer can then build its own customizable and private virtual network based on the cloud (i.e., CSPI-hosted) using the physical compute, memory, and networking resources provided by CSPI 200. As indicated earlier, these customer networks are referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as compute instances, on these customer VCNs. The compute instances can be in the form of virtual machines, bare-metal instances, and the like. CSPI 200 provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0117] In Figure 2 example embodiments depicted in FIG. 1, the physical components of CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches in the physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of a VCN. The compute instances can include virtual machine instances and bare-metal instances. For example, Figure 1 The various compute instances depicted in FIG. 1 can be hosted by Figure 2 The physical host machines depicted in FIG. 1. Virtual machine compute instances in a VCN can be executed by one host machine or multiple different host machines. The physical host machines can also host virtual host machines, container-based hosts, or functions, and the like. Figure 1 The VNICs and VCN VRs depicted in FIG. 1 can be executed by Figure 2 The NVDs depicted in FIG. 1. Figure 1 The gateways depicted in FIG. 1 can be executed by Figure 2 The host machines and / or NVDs described in FIG. 1.

[0118] A host machine or server can execute a hypervisor (also referred to as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances can be created, executed, and managed by the hypervisor on the host machine. The hypervisor on the host machine enables the physical computing resources (e.g., compute, memory, and network resources) of the host machine to be shared among various compute instances executed by the host machine.

[0119] For example, as depicted in Figure 2 , host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Generally, a hypervisor is a process or software layer that sits on top of an operating system (OS) of a host machine, which in turn executes on a hardware processor of the host machine. The hypervisor provides a virtualized environment by enabling the physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) of the host machine to be shared among various virtual machine compute instances executed by the host machine. For example, in Figure 2 , hypervisor 260 can sit on top of the OS of host machine 202 and enable the computing resources (e.g., processing, memory, and network resources) of host machine 202 to be shared among compute instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own operating system (referred to as a guest operating system), which can be the same as or different from the OS of the host machine. The operating system of a virtual machine executed by a host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor enables multiple operating systems to be executed simultaneously while sharing the same computing resources of the host machine. Figure 2 The host machines depicted in

[0120] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2 , compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0121] In some cases, an entire host machine can be provisioned to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by the host machine all belong to the same customer. In other cases, a host machine can be shared among multiple customers (i.e., multiple tenants). In such multi-tenancy scenarios, a host machine can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs of different customers. In certain embodiments, a bare metal compute instance is hosted by a bare metal server without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0122] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of data packets or frames to and from the compute instance. A VNIC is associated with a compute instance at the time the compute instance is created. In certain embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 embodiment depicted in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0123] For a compute instance hosted by a host machine, an NVD connected to the host machine also executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment depicted in Figure 2 FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 can also execute one or more VCN VRs 283 corresponding to VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0124] The host machines can include one or more network interface cards (NICs) that enable the host machines to connect to other devices. The NICs on the host machines can provide one or more ports (or interfaces) that enable the host machines to communicatively connect to another device. For example, the host machines can connect to the NVDs using one or more ports (or interfaces) provided on the host machines and on the NVDs. The host machines can also connect to other devices, such as another host machine.

[0125] For example, in Figure 2 , the host machine 202 connects to the NVD 210 using a link 220 that extends between a port 234 provided by a NIC 232 of the host machine 202 and a port 236 of the NVD 210. The host machine 206 connects to the NVD 212 using a link 224 that extends between a port 246 provided by a NIC 244 of the host machine 206 and a port 248 of the NVD 212. The host machine 208 connects to the NVD 212 using a link 226 that extends between a port 252 provided by a NIC 250 of the host machine 208 and a port 254 of the NVD 212.

[0126] The NVDs, in turn, connect to top-of-rack (TOR) switches via communication links, which connect to a physical network 218 (also referred to as a switch fabric). In certain embodiments, the links between the host machines and the NVDs, and between the NVDs and the TOR switches, are Ethernet links. For example, in Figure 2 , the NVDs 210 and 212 connect to the TOR switches 214 and 216 using links 228 and 230, respectively. In certain embodiments, the links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs that connect to the TORs is sometimes referred to as a rack.

[0127] The physical network 218 provides a communication fabric that enables the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In certain implementations, the physical network 218 is a multi-tier Clos network of switches, where the TOR switches 214 and 216 represent leaf-level nodes of the multi-tier and multi-node physical switch network 218. Different Clos network configurations are possible, including but not limited to 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and generally “n”-tier networks. An example of a Clos network is depicted in Figure 5 and described below.

[0128] Various different connection configurations are possible between the host machines and the NVDs, such as one-to-one configurations, many-to-one configurations, one-to-many configurations, and the like. In one-to-one configuration implementations, each host machine connects to its own separate NVD. For example, in Figure 2 In a one-to-one configuration, one host machine is connected to one NVD. For example, in Figure 2 In a one-to-many configuration, one host machine is connected to multiple NVDs.

[0129] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 An example is shown within CSPI 300 in which a host machine is connected to multiple NVDs. As shown in Figure 3 As shown in the example of FIG. 3, host machine 302 includes a network interface card (NIC) 304 that includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 can be Ethernet ports and links 320 and 322 between host machine 302 and NVDs 310 and 312 can be Ethernet links. NVD 310 is in turn connected to a first TOR switch 314 and NVD 312 is connected to a second TOR switch 316. Links between NVDs 310 and 312 and TOR switches 314 and 316 can be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices in a multi-layer physical network 318.

[0130] Figure 3 The arrangement depicted in

[0131] In the configuration depicted in Figure 3 In the configuration depicted in

[0132] Returning to Figure 2NVDs are physical devices or components that perform one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPUs, network processing units (NPUs), FPGAs, packet processing pipelines, etc.), memory (including cache), and ports. Various virtualization functions can be performed by software / firmware executed by one or more processing units of the NVD.

[0133] NVDs can be implemented in a variety of different forms. For example, in certain embodiments, NVDs are implemented as interface cards or smartNICs with on-board embedded processors, which are referred to as smartNICs. A smartNIC is a device that is separate from the NIC on a host machine. In Figure 2 In particular embodiments, NVDs 210 and 212 can be implemented as smartNICs that are connected to host machine 202 and to host machines 206 and 208, respectively.

[0134] However, smartNICs are just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, NVDs or one or more functions performed by NVDs can be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI 200. For example, an NVD can be implemented in a host machine, where functions performed by the NVD are performed by the host machine. As another example, an NVD can be part of a TOR switch, or a TOR switch can be configured to perform functions performed by an NVD, which enables the TOR switch to perform various complex packet transformations for a public cloud. A TOR that performs functions of an NVD is sometimes referred to as a smartTOR. In other implementations in which virtual machine (VM) instances are provided to customers instead of bare metal (BM) instances, functions performed by an NVD can be implemented inside a hypervisor of a host machine. In some other implementations, some of the functions of an NVD can be offloaded to a centralized service running on a set of host machines.

[0135] In certain embodiments, such as when implemented as a smartNIC as shown in Figure 2 In particular embodiments, an NVD can include a plurality of physical ports that enable it to be connected to one or more host machines and one or more TOR switches. Ports on an NVD can be classified as host-facing ports (also referred to as “southbound ports”) or network-facing or TOR-facing ports (also referred to as “northbound ports”). Host-facing ports of an NVD are ports used to connect the NVD to a host machine. Figure 2 Examples of host-facing ports in FIG. 2 include ports 236 on NVD 210 and ports 248 and 254 on NVD 212. Network-facing ports of an NVD are ports used to connect the NVD to a TOR switch. Figure 2 Examples of network-facing ports include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2A, NVD 210 is connected to TOR switch 214 using link 228 that extends from port 256 of NVD 210 to TOR switch 214. Similarly, NVD 212 is connected to TOR switch 216 using link 230 that extends from port 258 of NVD 212 to TOR switch 216. Figure 2

[0136] NVDs receive packets and frames from host machines via host-facing ports (e.g., packets and frames generated by compute instances hosted by the host machines) and, after performing necessary packet processing, can forward the packets and frames to TOR switches via the network-facing ports of the NVDs. NVDs can receive packets and frames from TOR switches via the network-facing ports of the NVDs and, after performing necessary packet processing, can forward the packets and frames to host machines via the host-facing ports of the NVDs.

[0137] In certain embodiments, there can be multiple ports and associated links between a NVD and a TOR switch. These ports and links can be aggregated to form a link aggregation group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between a NVD and a TOR switch) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in a LAG fails, then traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load balanced among the multiple physical links of a LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be located between a NVD and a TOR switch, between a host machine and a NVD, and so on.

[0138] ​NVDs implement or perform network virtualization functions. These functions are performed by software / firmware executed by the NVDs. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions to create VCN networks; functions to implement network policies, such as VCN security list (firewall) functionality; functions to facilitate packet routing and forwarding to and from compute instances in a VCN; and so on. In certain embodiments, upon receiving a packet, the NVD is configured to execute a packet processing pipeline to process the packet and determine how to forward or route the packet. As part of this packet processing pipeline, the NVD can perform one or more virtual functions associated with an overlay network, such as executing a VNIC associated with a compute instance in a VCN, executing a virtual router (VR) associated with a VCN, encapsulation and decapsulation of packets to facilitate forwarding or routing in a virtual network, execution of certain gateways (e.g., local peering gateway), implementation of security lists, network security groups, network address translation (NAT) functionality (e.g., translating public IPs to private IPs on a per-host basis), throttling functionality, and other functions.

[0139] In certain embodiments, the packet processing datapath in an NVD can include multiple packet pipelines, each composed of a series of packet transformation stages. In certain implementations, upon receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed in a linear fashion, one stage after another, until the packet is either dropped or sent out through an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., validate headers, enforce throttling, insert new Layer 2 headers, enforce L4 firewall, VCN encapsulation / decapsulation, etc.) so that new pipelines can be constructed by combining existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0140] The NVDs can perform control plane and data plane functions corresponding to the control plane and data plane of a VCN. Examples of a VCN control plane are also depicted in Figure 14 , Figure 15 , Figure 16 and Figure 17 (see reference numbers 1416, 1516, 1616, and 1716) and described below. Examples of a VCN data plane are depicted in Figure 14 , Figure 15 , Figure 16 and Figure 17Control plane functions include functions to configure the network on how control data is forwarded (e.g., set routes and routing tables, configure VNICs, etc.). In certain embodiments, a VCN control plane is provided that computes all overlay-to-substrate mappings and publishes them to NVDs and virtual network edge devices (such as various gateways, such as DRG, SGW, IGW, etc.). Firewall rules can also be published using the same mechanism. In certain embodiments, NVDs only get mappings relevant to that NVD. Data plane functions include functions to actually route / forward data packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating network packets of a customer before they traverse the substrate network. The encapsulation / decapsulation functionality is implemented on NVDs. In certain embodiments, NVDs are configured to intercept all network packets going to and from host machines and perform network virtualization functions.

[0141] As indicated above, NVDs perform various virtualization functions, including VNICs and VCN VRs. An NVD can perform a VNIC associated with a compute instance hosted by one or more host machines connected to the VNIC. For example, as depicted in Figure 2 As depicted in the middle, NVD 210 performs functionality of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host machine 206, and performs VNIC 284 associated with compute instance 274 hosted by host machine 208. A host machine can host compute instances belonging to different VCNs (belonging to different customers), and an NVD connected to the host machine can perform VNICs corresponding to the compute instances (i.e., perform VNIC-related functionality).

[0142] An NVD also performs a VCN virtual router corresponding to a VCN of a compute instance. For example, in the embodiment depicted in Figure 2 NVD 210 performs VCN VR 277 corresponding to the VCN that compute instance 268 belongs to. NVD 212 performs one or more VCN VRs 283 corresponding to one or more VCNs that the compute instances hosted by host machines 206 and 208 belong to. In certain embodiments, a VCN VR corresponding to a VCN is performed by all NVDs connected to host machines that host at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, then NVDs connected to that host machine can perform VCN VRs corresponding to those different VCNs.

[0143] In addition to VNICs and VCN VRs, NVDs can also execute various software (e.g., daemons) and include one or more hardware components that facilitate various network virtualization functions performed by the NVDs. For simplicity, these various components are grouped together as the “packet processing components” shown in Figure 2 The packet processing components for an NVD can include a packet processor configured to interact with the ports and hardware interfaces of the NVD to monitor all packets received and transmitted by the NVD and store network information. The network information can include, for example, network flow information identifying different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In certain embodiments, the network flow information can be stored on a per-VNIC basis. The packet processor can perform per-packet manipulations and implement stateful NAT and L4 firewall (FW). As another example, the packet processing components can include a replication agent configured to replicate information stored by the NVD to one or more different replication target repositories. As yet another example, the packet processing components can include a logging agent configured to perform logging functions for the NVD. The packet processing components can also include software for monitoring the performance and health of the NVD and possibly also the status and health of other components connected to the NVD.

[0144] Figure 1 Components of an example virtual or overlay network are shown, including a VCN, a subnet within the VCN, compute instances deployed on the subnet, VNICs associated with the compute instances, a VR for the VCN, and a set of gateways configured for the VCN. Figure 1 The overlay components depicted in Figure 2 may be executed or hosted by one or more of the physical components depicted in Figure 2 For example, a compute instance in a VCN can be executed or hosted by one or more host machines depicted in For example, certain gateways can be executed by a smartNIC, while other gateways can be executed by one or more host machines or other implementations of NVDs.

[0145] As described above, a compute instance in a customer VCN can communicate with various different endpoints, where the endpoints can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or with endpoints that are outside the VCN of the source compute instance. These communications are facilitated using the VNICs associated with the compute instances, the VCN VRs, and gateways associated with the VCNs.

[0146] For communications between two compute instances on the same subnet in a VCN, the communications are facilitated using the VNICs associated with the source and destination compute instances. The source and destination compute instances can be hosted by the same host machine or different host machines. A packet originating from the source compute instance can be forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which can include execution of the VNIC associated with the source compute instance. Since the destination endpoint for the packet is located within the same subnet, execution of the VNIC associated with the source compute instance causes the packet to be forwarded to the NVD executing the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can execute on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs can use the routing / forwarding table stored by the NVD to determine the next hop for the packet.

[0147] For a data packet that is to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the data packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the data packet is processed using a data packet processing pipeline, which can include the execution of one or more VNICs and a VR associated with the VCN. For example, as part of the data packet processing pipeline, the NVD executes or invokes functionality corresponding to the VNIC associated with the source compute instance (also referred to as executing the VNIC). The functionality executed by the VNIC can include looking at the VLAN tag on the data packet. Since the destination of the data packet is outside the subnet, the VCN VR functionality is next invoked and executed by the NVD. The VCN VR then routes the data packet to an NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the data packet and forwards the data packet to the destination compute instance. The VNICs associated with the source and destination compute instances can be executed on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0148] If the destination for the data packet is outside the VCN of the source compute instance, then the data packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to an NVD connected to that host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the data packet is outside the VCN, the data packet is subsequently processed by the VCN VR for that VCN. The NVD invokes the VCN VR functionality, which results in the data packet being forwarded to an NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within a customer’s on-premise network, then the data packet can be forwarded by the VCN VR to an NVD executing a DRG gateway configured for the VCN. The VCN VR can be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance, or by a different NVD. The gateway can be executed by an NVD, which can be a smartNIC, a host machine, or other NVD implementation. The data packet is then processed by the gateway and forwarded to the next hop, which facilitates the communication of the data packet to its intended destination endpoint. For example, in Figure 2In the depicted embodiment, a data packet originating from compute instance 268 can be transmitted from host machine 202 to NVD 210 over link 220 (using NIC 152). On NVD 210, VNIC 276 is invoked as it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to inspect the information encapsulated in the data packet and determine the next hop for forwarding the data packet with the goal of facilitating the transmission of the data packet to its intended destination endpoint, and then forward the data packet to the determined next hop.

[0149] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 200 can include instances in the same VCN or other VCNs, which can be a customer’s VCNs, or VCNs that do not belong to a customer. Communication between endpoints hosted by CSPI 200 can be performed over physical network 218. Compute instances can also communicate with endpoints that are not hosted by CSPI 200 or are external to CSPI 200. Examples of these endpoints include endpoints within a customer’s on-premise network or data center, or public endpoints that are accessible over a public network, such as the Internet. Communication with endpoints external to CSPI 200 can be performed over a public network (e.g., the Internet) using various communication protocols (not shown in Figure 2 or a private network (not shown in Figure 2 ).

[0150] Figure 2 The architecture of CSPI 200 depicted in FIG. 1 is merely an example and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI 200 can have more or fewer systems or components than those shown in FIG. 1, two or more systems can be combined, or there can be a different configuration or arrangement of the systems. Figure 2 Figure 2 The systems, subsystems, and other components depicted in FIG. 1 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software can be stored on non-transitory storage media (e.g., memory devices).

[0151] Figure 4 A connection between a host machine and an NVD is depicted for providing I / O virtualization to support multi-tenancy in accordance with certain embodiments. As Figure 4 ​As depicted, host machine 402 executes a hypervisor 404 that provides a virtualized environment. Host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is attached to a VNIC executed by NVD 412. Figure 4 In one embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0152] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416 and VM2 408 is attached to logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, each tenant's virtual machines believe they have their own host machine and NIC.

[0153] In some embodiments, each logical NIC is assigned its own VLAN ID. Therefore, a specific VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, the hypervisor appends a tag assigned to tenant #1 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Similarly, when a packet is transmitted from VM2 408, the hypervisor appends a tag assigned to tenant #2 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Accordingly, the packet 424 transmitted from host machine 402 to NVD 412 has an associated tag 426 identifying the specific tenant and associated VM. On the NVD, for a data packet 424 received from the host machine 402, the tag 426 associated with the data packet is used to determine whether the data packet is processed by VNIC-VM1 420 or VNIC-VM2 422. The data packet is then processed by the corresponding VNIC. Figure 4 The configuration described in [the document] enables each tenant's compute instance to believe that they own their own host machine and NIC. Figure 4 The setup described in [the document] provides I / O virtualization to support multi-tenancy.

[0154] Figure 5A simplified block diagram of a physical network 500 is depicted in accordance with certain embodiments. Figure 5 The depicted embodiment is structured as a Clos network. A Clos network is a particular type of network topology designed to provide connectivity redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-stage or multi-tier switching network, where the number of stages or tiers can be two, three, four, five, etc. Figure 5 The depicted embodiment is a 3-tier network, including tier 1, tier 2, and tier 3. TOR switches 504 represent tier 0 switches in the Clos network. One or more NVDs are connected to the TOR switches. Tier 0 switches are also referred to as edge devices of the physical network. Tier 0 switches are connected to tier 1 switches, also referred to as leaf switches. In turn, tier 1 switches are connected to tier 2 switches, also referred to as spine switches. Tier 2 switches are connected to tier 3 switches, also referred to as super-spine switches. The communication of data packets over the physical network 500 is typically performed using one or more tier 3 communication protocols. Typically, all tiers of the physical network (except the TOR tier) are n-way redundant, thus allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, thus enabling scaling of the physical network. Figure 5 In the depicted embodiment, a set of "n" tier 0 TOR switches are connected to a set of "n" tier 1 switches and form a pod. Each tier 0 switch in a pod is interconnected to all tier 1 switches in that pod, but there is no connectivity of switches between pods. In certain implementations, two pods are referred to as a block. Each block is served by or connected to a set of "n" tier 2 switches (sometimes referred to as backbone switches). There can be several blocks in the physical network topology. Tier 2 switches, in turn, are connected to "n" tier 3 switches (sometimes referred to as super-spine switches). The communication of data packets over the physical network 500 is typically performed using one or more tier 3 communication protocols. Typically, all tiers of the physical network (except the TOR tier) are n-way redundant, thus allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, thus enabling scaling of the physical network.

[0155] A feature of a Clos network is that the maximum number of hops from one tier 0 switch to another tier 0 switch (or from an NVD connected to a tier 0 switch to another NVD connected to a tier 0 switch) is fixed. For example, in a 3-tier Clos network, a data packet arriving from one NVD to another NVD requires at most seven hops, where the source and target NVDs are connected to the leaf tier of the Clos network. Likewise, in a 4-tier Clos network, a data packet arriving from one NVD to another NVD requires at most nine hops, where the source and target NVDs are connected to the leaf tier of the Clos network. Thus, the Clos network architecture maintains consistent latency throughout the network, which is important for intra-data center and inter-data center communications. The Clos topology is horizontally scalable and cost effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches at various tiers (e.g., more leaf switches and backbone switches) and by increasing the number of links between switches of adjacent tiers.

[0156] In certain embodiments, each resource within a CSPI is assigned a unique identifier referred to as a cloud identifier (CID). This identifier is included as part of the information for the resource and can be used to manage the resource, e.g., via a console or through an API. An example syntax for a CID is:

[0157] ocid1.<RESOURCE TYPE>. <realm>.[REGION][.FUTURE USE].<UNIQUE ID>

[0158] wherein,

[0159] ocid1: a literal string indicating the version of the CID;

[0160] resource type: the type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.);

[0161] realm: the realm in which the resource resides. Example values are "cl" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal government cloud realm, etc. Each realm can have its own domain name;

[0162] region: the region in which the resource resides. This section can be empty if the region does not apply to the resource;

[0163] future use: reserved for future use.

[0164] unique ID: the unique portion of the ID. The format can vary by the type of resource or service.

[0165] Packet-level data-centric protection enforcement

[0166] Figure 6 is a simplified block diagram of an environment 600 that illustrates performing packet-level data-centric protection enforcement according to certain embodiments. Environment 600 includes a plurality of systems that are communicatively coupled to one another. Figure 6 The systems in include a zero trust access (ZTA) service 602, enforcement points 610 (e.g., EPs 610A1-610AN, EPs 610B1-610BN, gateway 630), computing devices 620, gateway 630, and tenancy 640.

[0167] ZTA service 602 includes a distribution engine 604, a rules engine 606, and a data store 608. While distribution engine 604, rules engine 606, and data store 608 are illustrated as part of ZTA service 602, one or more of these components can be external to ZTA service 602. Computing devices 620, which can be referred to herein as "servers 620" or "server computing devices 620," can include a hypervisor (HV) (not shown) that can host virtual machines (VMs).

[0168] Figure 6 The environment 600 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some embodiments, environment 600 may have a greater... Figure 6 The system or components shown may be more or fewer, may be combined into two or more systems, or may have different system configurations or arrangements. Figure 6 The systems, subsystems, and other components described herein may be implemented using software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the respective systems. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0169] exist Figure 6 The embodiments described herein illustrate novel techniques for enforcing packet-level, data-centric protection. Compared to prior art that focuses on rules specifying which resources can access data, the assertion concerns described herein concern what can be done about the data itself. In other words, assertion / policy statements concern how data can flow through one or more networks. As an example, an assertion / policy statement could simply be "Red data never leaves my lease," "Blue data never reaches the Internet," "Blue data is not stored with red data," "Green data never leaves data zone 2," and so on. In some configurations, policies may also include rules / assertions / statements specifying how data moves between resources and / or resources within data zones. In some examples, data policies may reference data zones, data attributes, and local conditions or circumstances. While example policy language definitions are discussed herein, other policy languages ​​may also be used in conjunction with the techniques described herein.

[0170] Using the techniques described herein, security rules associated with policy 616 can protect the flow of data through one or more networks by using the source and destination of the data. According to some examples, permission checks are shifted from runtime to compile time, thus saving time as data flows through one or more networks. As briefly discussed above, after policy 616 is created, its rules are enforced across one or more networks by EPs (such as EP 610) and gateways 630 distributed at different network points across the one or more networks.

[0171] In some examples, the user does not need to specify what devices / components to use to enforce the policy, or where those devices are located. Rather, the ZTA service 602 or some other component can deploy and / or instruct different EPs 610 already located within the network to enforce the policy. In some examples, the EP functionality 610 can be deployed on all or a portion of the network devices involved in data transfer within the network(s). For example, the EP 610 can be deployed on gateways 630, smartNICs, gateways 630, and other types of network devices / components.

[0172] According to some configurations, the ZTA service 602 communicates with different EPs 610 within the network. In some examples, the ZTA service 602 provides enforcement data 612 to the different EP(s) 610 within the network indicating how the EP should enforce rules associated with one or more policies 616. Individual EPs 610 can also communicate with the ZTA service 602 to request enforcement data 612. In some configurations, different EPs 610 can enforce different rules / evaluations. For example, labels and attributes (e.g., names) can be evaluated at layer 4 by a smartNIC EP, and L7 attributes (including but not limited to: path, request cookie, request header, URL query, request method, country, source IP address, destination IP address) can be evaluated at a target service or egress proxy (e.g., gateway).

[0173] As discussed herein, virtual networks are implemented using software virtualization techniques (e.g., hypervisors, virtualization functionality implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, smart TORs implementing one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of a physical network. For example, in certain embodiments, NVDs are implemented as interface cards referred to as smartNICs or smartNICs with on-board embedded processors.

[0174] A smartNIC is a device that is separate from the NIC on a host machine, such as a compute device 620 that hosts instances. According to some configurations, a smartNIC within a network includes functionality that operates as an EP 610. Other NVDs can also include EP 610. For example, in some other implementations, the NVD or one or more functions performed by the NVD can be incorporated into or performed by one or more compute devices 620, one or more TOR switches, and other components of CSPI. For example, in the case of provisioning virtual machine (VM) instances to customers instead of bare metal (BM) instances, the functionality performed by the NVD can be implemented inside the hypervisor of the host machine. In some other implementations, some of the functionality of the NVD can be offloaded to a centralized service running on a set of host machines.

[0175] In some examples, upon receiving a data packet, the EP 610 can be configured to execute a data packet processing pipeline to process the data packet and determine how to process the data packet. As part of this data packet processing pipeline, the EP 610 can perform one or more virtual functions, such as encapsulation and decapsulation of data packets, identification of origin IDs, etc., to facilitate processing policies / rules and other functions. In some configurations, a data packet processing datapath in a device that includes the EP 610 can include multiple data packet pipelines, each pipeline consisting of a series of data packet transformation stages. According to some examples, upon receiving a data packet, the data packet can be processed in a linear fashion, one stage after another, until the data packet is either dropped or sent out through an interface associated with the EP 610.

[0176] Figure 6 The gateway 630 shown in FIG. 6 can be any type of gateway such as, but not limited to, a dynamic routing gateway (DRG), an internet gateway (IGW), a network address translation (NAT) gateway, a local peering gateway (LPG), a service gateway (SGW), etc. A DRG acts as a virtual router, providing a path for traffic between on-premise networks and VCNs, and can also be used to route traffic between VCNs. An IGW enables compute instances on a VCN to communicate with public endpoints that are accessible over a public network, such as the Internet. A NAT gateway can be configured for a customer’s VCN, which enables cloud resources in the customer’s VCN that do not have a dedicated public overlay IP address to access the Internet, and does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). An LPG is a gateway that can be added to a VCN and enables the VCN to peer with another VCN in the same region. An SGW can be configured for a VCN and provides a path for private network traffic between the VCN and service endpoints supported in the service network.

[0177] In Figure 6 In the depicted embodiment, the ZTA service 602 is configured to oversee, configure, monitor, and maintain the zero trust network infrastructure. In some examples, the zero trust network is a software-defined network operating on top of existing cloud infrastructure (further details in Figures 1-5 and Figures 14-18 ) that enables secure and policy-driven communication between resources (e.g., clients, services,...) of the network. According to some examples, the ZTA service 602 implements zero trust principles to help ensure that network interactions are authenticated, authorized, and encrypted, thereby enhancing security and access control. The zero trust network can span multiple environments, including public clouds, private data centers, and on-premises locations, thereby providing a flexible and secure network foundation for various applications and services.

[0178] According to some configurations, enforcement of policies is facilitated by augmenting virtual node packet processing. In some examples, an origin ID is assigned / associated to one or more different resources within the network. According to some examples, the origin ID is a unique 32-bit unsigned integer that identifies a particular data class in a customer’s network, possibly across multiple clouds or on-premises equipment. In other examples, the origin ID can be a simple distributed counter that increments with each new origin ID generation. As discussed above, the resources can be instances, services, or user principals within the network that can store or access data, such as but not limited to object storage buckets, compute instances, databases, functions, etc.

[0179] In some cases, the origin ID can be carried by the packet as it travels through the network until it reaches the destination. According to some configurations, each enforcement point 610 can use the origin ID to determine the portion of enforcement data 612 to apply to determine the allowed communication path for the packet before it is transmitted to the next hop in the network path to the destination.

[0180] In some examples, the origin ID can be used to indicate the source of the packet, and another origin ID can be used to indicate the target destination of the packet. According to some configurations, the ZTA service 602 collaborates with a data security control plane (DSCP) to assign origin IDs to each EP 610 in the network. The ZTA service 602 can also provide origin IDs to other resources (e.g., gateways and identity data plane) to enable them to evaluate security policies.

[0181] According to some examples, resources can be "labeled" to associate the resources with a name / classification. For example, data that is to be protected by policy can be labeled. According to some examples, resources can be labeled automatically and / or manually. For example, automated labeling can be performed to discover and label data, such as sensitive data. Sensitive data can need to be protected and comply with compliance frameworks, such as PCI DSS (Payment Card Industry Data Security Standard), HIPAA (Health Insurance Portability and Accountability Act), and GDPR (General Data Protection Regulation). In many cases, organizations initially know where sensitive data is located when creating new greenfield applications, projects, etc. However, over time, this sensitive data is moved for legitimate purposes (e.g., business analytics, software migration, and new development projects), making it difficult to protect this data using prior techniques.

[0182] In Figure 6 In examples, one or more EPs 610 can be configured for each zone 640 to perform processing related to asserting and data policies. This processing can include identifying relevant enforcement data 612 associated with a particular data communication or data movement request and determining whether the communication is allowed or disallowed. The EPs 610 can be physical components or can be logical or virtual components (e.g., implemented in software). In certain implementations, an EP 610 can be configured for a single resource or a group of resources that are sources and targets of data movement or communication. For example, for a compute instance (e.g., a virtual machine), a smartNIC (often a network virtualization device (NVD)) implementing a virtual network interface card (VNIC) for the compute instance can be configured as an EP 610 for data communications involving the compute instance.

[0183] In some examples, the decision whether data movement between two resources is allowed or disallowed is pre-computed / determined by the rules engine 606 based on policies and assertions. The rules engine 606 interprets policies and generates enforcement data 612 for rules specified in the policies 616 that are enforced by different EPs 610. In some examples, the rules engine generates a graph 614 that includes resources of a network and represents communications between different resources.

[0184] According to some configurations, the graph 614 describes possible data movements between devices or services modeled as vertices by edges of the graph. In some cases, an edge represents a policy or assertion outcome. To create this graph, the rules engine 606 determines the outcome / consequence of applying policy declarations in a policy for different connections between enforcement points (e.g., neighbors of an enforcement point). For example, by referencing a node in the graph (e.g., an enforcement point in a network), it can be easily determined which other nodes (e.g., other enforcement points) the node is allowed to communicate with.

[0185] In some examples, each EP 610 uses enforcement data 612 to determine how data should flow between itself and neighboring resources and other EPs 610. In some cases, the enforcement data 612 is specific to each EP 610. According to some examples, the rules engine 606 rebuilds the graph 614 and updates the enforcement data 612 in response to changes to the policies 616. The graph 614 and / or the enforcement data 612 can be automatically updated in response to other conditions, such as but not limited to the addition of new data, the addition / removal of resources, changes to policies, changes to assertions, the addition / removal of zones, etc.

[0186] According to some configurations, the distribution engine 604 distributes the enforcement data 612 to the different EPs 610 (e.g., vertices in the graph) that can enforce the rules associated with the policies 616. At each network hop, the EP 610 accesses the enforcement data 612 received from the distribution engine 603 to determine how to transmit a received data packet. Generally speaking, when an EP 610 receives a data packet, the EP 610 consults the enforcement data 612 that identifies pre-computed decisions to determine whether to permit communication of a particular data between a source resource (which has the data) and a target resource (which requests the particular data). In other words, the EP 610 that receives the data packet uses the enforcement data 612 to ensure that the data packet is only forwarded along permitted paths. Additionally, unlike existing network security policies, the ZPR architecture allows for specifying lease range policies that can span across network and zone boundaries.

[0187] Referring to Figure 6 , a simple example is illustrated. Referring to Figure 6 , a first zone 650A and a second zone 650B are illustrated within the lease 640. As discussed above, a zone 650 includes a collection of resources that have similar data classification attributes. In some examples, a zone 650 can default to preventing data from being transmitted outside of the zone, or a zone 650 can default to allowing data to be transmitted outside of the zone.

[0188] In some configurations, one or more assertions can be configured for each zone 650. An assertion defined for a particular data zone is a set of one or more rules that apply to a set of resources classified into or belonging to the particular data zone and that cannot be violated. An assertion is an expression that governs access to data within a zone. In the current example, the zone 650A does not include any assertions, and the zone 650B includes an assertion 632B that specifies "Zone 650B can transmit data where all { source.DataCategory.attribute!= "PII"}." The rule made by the assertion 632B is that any data other than "PII data" can be transmitted from the zone 650B.

[0189] As briefly discussed above, one or more data policies 616 can be defined between two zones 650. A data policy between two data zones is a rule for data movement between resources classified to the two data zones. A data policy can reference a data zone, a data attribute, and a local situation or condition. In the current example, a single policy is specified between zone 650A and zone 650B, which states "Allow zone 650A to transfer data to zone 650B."

[0190] Referring to zone 650A (which in this example performs data processing), there is a first instance 660A that has been assigned an origin ID "2194867289" and is associated with a first attribute "Blue," and a second instance 660B that has been assigned an origin ID "4194867289" and is associated with the first attribute "Blue." Referring to zone 650B (which in this example is a data sink), storage 624A, which has been assigned an origin ID "4214867299," can see data 626A associated with a first attribute "Red" and a second attribute "PII." Storage 624B, which has been assigned an origin ID "3294867299," and data 626B is associated with the first attribute "Red."

[0191] In the current example, the rules engine 606 accesses the policies, attributes, and assertions to generate enforcement data 612 for each EP 610. For purposes of explanation, assume that instance 660A, instance 660B, storage 624A, and storage 624B are configured as EPs 610. According to some configurations, the rules engine 606 generates a graph 614 that includes a node for each EP 610. The graph 614 illustrates possible data movement between instance 660A, instance 660B, storage 624A, and storage 624B, which are modeled as vertices, through edges of the graph. In some cases, an edge can represent a policy or assertion conclusion.

[0192] To create the graph, the rules engine 606 determines all results / conclusions of the policy statements in the policies that apply for different connections between the EPs (instance 660A, instance 660B, storage 624A, and storage 624B in this example). In some configurations, the determination of the results is performed at compile time (before a data request is made), rather than at runtime when a data request is made, saving time as data flows through one or more networks. The rules engine 606 generates enforcement data 612 for each EP 610 based on the graph 614 associated with the policies. In some examples, a user can view the graph to easily determine connected resources and which resources are neighbors of other resources.

[0193] As an example, the rules engine 606 computes the expected outcomes between each of the different connections (in this example, instance 660A, instance 660B, storage 624A, and storage 624B). The following table illustrates an example of the vertex and edge relationships at compile time.

[0194]

[0195]

[0196]

[0197] In this example, this determined information can be used to generate enforcement data 612, which can be distributed by the distribution engine 604 to each EP 610 (e.g., instance 660A, instance 660B, storage 624A, and storage 624B in this example). The enforcement data is essentially a pre-defined instruction for each EP that specifies how the EP should handle data requests from any adjacent nodes based on the set of pre-evaluated policies associated with its origin ID. Similarly, the enforcement data specifies how to handle the transmission of outgoing data packets to any adjacent enforcement points or service endpoints defined in the graph based on the pre-evaluated assertions associated with its origin ID.

[0198] As shown, the enforcement data 612 of this example includes the origin ID of the source and the origin ID of the possible destination, as well as an indicator that identifies whether the transfer / delivery of data is allowed. In some examples, an outcome value of "0" indicates that the transfer / delivery is not allowed, with a value of "1" indicating that the transfer / delivery is allowed.

[0199]

[0200]

[0201] According to some examples, rather than passing all enforcement data 612 to each EP 610, the distribution engine 604 provides each EP 610 with enforcement data 612 relevant to that particular EP 610’s enforcement. Referring to the table above, the rules engine 606 and / or the distribution engine 604 can extract a unique set of instructions applicable to each origin ID. For example, the following enforcement data 612 describes the instructions for data source 3294867299 (storage 624B) that indicates all possible connections for that data source at compile time: 4194867289: 1 2194867289: 1. While not illustrated, the rules engine 606 and / or the distribution engine 604 provide enforcement data 612 specific to each resource. According to some examples, each EP 610 uses the enforcement data 612 to determine how data should flow between itself and adjacent resources and other EPs 610. In some examples, the enforcement data 612 is specific to each EP 610.

[0202] For purposes of explanation, assume that instance 660A requests data 626A from storage 624A and requests data 626B from storage 624A. In some examples, in response to a request to transfer data from a first entity in one zone 650 to a second entity in a second zone 650, any assertions configured for the zone are identified and evaluated by the EP 610. Evaluation of the assertions results in allow / deny directives. Only when no assertions of the zone are violated is the requested communication of data allowed. Some example assertions that can be configured for a data zone include, but are not limited to, not allowing certain types of data to leave the zone, not sending data from the zone to a computer connected to the internet, not allowing communication with hosts that are more than x hops away, etc.

[0203] In some examples, for faster data propagation and cache efficiency, an edge cache proxy (not shown) can be used with the ZTA service 602 to manage fetching and making available a subset of the instructions relevant to a given EP 610. According to some examples, the task of generating enforcement data 612 for each EP 610 can be parallelized, as each EP 610 represents an independent unit of work. This means that whenever an end user updates a policy 616, modifies zone 650 membership, or changes an assertion 632, the rules engine 612 can recalculate the enforcement data 612 associated with the subset of resources affected. Subsequently, this updated enforcement data 612 can be provided to the edge cache closest to the EP 610, thereby helping to ensure that the enforcement data 612 is available to the EP 610 in a timely manner.

[0204] Example algorithm

[0205] According to some examples, the enforcement data 612 can be loaded into the memory of the EP 610 using a Trie algorithm. This helps reduce memory, while also being time efficient. Using this technique, the EP 610 will use the data provided in its respective origin ID mapping to construct a Trie data structure to efficiently evaluate requests at the data source.

[0206] For example, assume that the rules engine 606 is evaluating data security assertions and policies for a resource associated with origin ID '1000000001'. We can use a set of 32-bit unsigned integers that represent the origin IDs of the devices that data can move to. An example of the enforcement data 612 for the resource idl.objectstore.region.tenancy.namespace.bucketname.objectname identified by origin ID: 1000000001 can be: 4194867299:0, 4194867289:1, 4214867299:0, 3294867299:1, 3194867299:1, 3394867299:0, 3494867299:1, 3794867299:1.

[0207] The source EP for the EP enforcement data for each data packet can be visualized using a Trie. The EP 610 first looks up the origin ID IP option / header in the incoming data packet, then it checks its latest cache and inserts it into a Trie data structure. This lookup process implicitly goes from the data source origin ID to all possible target origin IDs with a predetermined outcome. Referring to the diagram 614A shown in Figure 6 Each node in the above example represents an origin ID taken from the IP options of a target data packet as a 32-bit unsigned integer. The path from the root to the leaf node corresponds to the target origin ID inserted as an unsigned integer. The value in brackets at the leaf node indicates the associated bit value (1 or 0). Where 0 means the request being evaluated is rejected, and 1 means it is allowed.

[0208] For example: Insert origin ID 4194867299, which is associated with bit 0 <-- if a request from 4194867299 requests data from source origin ID '1000000001', then the connection will be dropped. In other examples, origin IDs that evaluate to '0' can not be stored, reducing the instruction size by assuming that all items not in the list will evaluate to '0'. Insert origin ID 4194867289, which is associated with bit 1 <-- if a request from 4194867289 requests data from source origin ID '1000000001', then the connection will be allowed. Insert origin ID 4214867299, which is associated with bit 0 <-- if a request from 4214867299 requests data from source origin ID '1000000001', then the connection will be dropped. Insert origin ID 3294867299, which is associated with bit 1 <-- if a request from 3294867299 requests data from source origin ID '1000000001', then the connection will be allowed. Insert origin ID 3194867299, which is associated with bit 1 <-- if a request from 3194867299 requests data from source origin ID '1000000001', then the connection will be allowed. Insert origin ID 3394867299, which is associated with bit 0 <-- if a request from 3394867299 requests data from source origin ID '1000000001', then the connection will be dropped. Insert origin ID 3494867299, which is associated with bit 1 <-- if a request from 3494867299 requests data from source origin ID '1000000001', then the connection will be allowed. Insert origin ID 3794867299, which is associated with bit 1 <-- if a request from 3794867299 requests data from our source origin ID '1000000001', then the connection will be allowed.

[0209] To perform a lookup, the EP associated with source '1000000001' can traverse the trie based on the digits of the 32-bit unsigned integer of the requestor's origin ID. For example: To look up 3294867299, you would follow the path 3 -> 2 -> 9 -> 4 -> 8 -> 6 -> 7 -> 2 -> 9 -> 9 and find the leaf node with value 1. To look up 3394867299, you would follow the path 3 -> 3 -> 9 -> 4 -> 8 -> 6 -> 7 -> 2 -> 9 -> 9 and find the leaf node with value 0.

[0210] According to some examples, the use of SYN packets can be used to reduce the overall lookup / dictionary tree origin ID instruction search frequency in a given TCP / IP session. In TCP / IP socket connection establishment, there is a special type of packet called a "SYN" packet, which is short for "synchronize." SYN packets are used as part of the TCP three-way handshake during connection establishment between two devices. The client sends a SYN, the server sends a SYN-ACK, and the client sends an ACK.

[0211] The following is example pseudo code:

[0212]

[0213] As described above, origin IDs can be used to identify and label resources, such as data assets. Relationships between data identified via origin IDs can be used to construct a graphical model of the data administrator's intent. In some examples, the graph identifies security relationships between a given data asset and various data requesters (e.g., instances, services, on-premise services, or other endpoints within the cloud with their own advertised origin IDs). Thus, embodiments provide the ability to generate a graphical blueprint model of the data administrator's data security intent.

[0214] According to some examples, this security and compliance relationship between data assets and requesters can be represented as a graph. Time snapshots of the graph vertices and edges can be backed up into storage. Once various graph snapshots have been collected over time, the differences between them can identify any changes in the security posture. These changes in security posture can be used to understand the level of compliance with respect to various industry standards or contractual agreements. This provides the ability to provide compliance reports on a customer's data security posture based on the increments between the graphical blueprint models of the security intent generated and saved over time.

[0215] Figure 7 is a simplified block diagram of an environment 700 illustrating ZTA service 602 interacting with control plane(s) and data plane(s) in accordance with certain embodiments. Environment 700 includes a plurality of systems communicatively coupled to one another. Figure 7 The systems in include ZTA 602, device 622, control plane(s) 702, and data plane(s) 704.

[0216] Figure 7 The environment 700 depicted in is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the environment 700 can have more or fewer systems than those shown in Figure 7 More or fewer systems or components than those shown in FIG. 6 can be used, two or more systems can be combined, or different system configurations or arrangements can be used. Figure 7 The systems, subsystems, and other components depicted in FIG. 6 can be implemented with software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device).

[0217] As discussed above, the rules engine 606 accesses the user access policy 616 associated with the device 622 and generates enforcement data 612. The distribution engine 604 propagates the enforcement data 612 to a control plane across different regions (not shown). In some configurations, the control plane distributes this data to various data planes. In some configurations, the control plane(s) 702 (see below Figures 14-18 ) propagate the enforcement data 612 from the ZTA service 602 to the data plane(s) 704. According to some examples, the control plane 702 maps resources with their associated tag / origin ID. In some configurations, the distribution engine 604 propagates the enforcement data 612 to different regions (not pictured).

[0218] As briefly discussed above, in some configurations, the control plane 702 assigns origin IDs to tagged resources. For resources that are not visible to the control plane 702 within the network(s), a user can define resources that can be associated with an origin ID.

[0219] Figure 8 is a simplified diagram illustrating different components that can be included in a policy declaration / assertion according to an example.

[0220] According to some examples, the commands 802 include allow commands 804 and deny commands 806. The allow commands 804 specify to allow access to a resource. The deny commands 806 specify to deny access to a resource. The deny commands 806 can assist with delegation of duties, deny access to a resource, control the flow of traffic within one or more networks, etc.

[0221] The deny commands 806 explicitly deny permissions that are not covered by other policy declarations. The deny commands 806 follow the same syntax as the allow declarations. In some configurations, the deny commands 806 are evaluated before any allow commands (following the same tree scope behavior as the allow commands). The benefit of evaluating the deny commands 806 before the allow commands 804 is that the deny commands 806 allow a user to write delegation of policies to subtenants without worrying about the subtenants overriding the intent of the root administrator. In other words, a delegator can expand on allow commands, but they cannot override denies.

[0222] In some examples, one or more tags 820 can be used to tag a resource. For example, a user, a group of users, a computing resource, a group of computing resources, a storage device, a group of storage devices, data, and the like. In some configurations, tags and attributes (e.g., names) can be evaluated by the smartNIC EP at layer 4, and L7 attributes (including but not limited to: path, request cookie, request header, URL query, request method, country, source IP address, destination IP address) can be evaluated at a target service or egress proxy (e.g., gateway). Policy declarations can also use multiple tags 820.

[0223] Tags can be defined by a user, and / or tags can be automatically created for different resources. If a single tag appears before the base name, the tag is written before the base name, with no additional syntax (except for a separating space). Examples: "red network", "region: Europe network", "'big shot' user", "location: 'New York' user". If multiple tags appear before the base name, the tags can be written before the base name, separated by commas, in order. Examples: "approved, red network", "approved, blue, region: Europe network", "authorized, 'big shot' user", "authorized, department: sales, location: 'New York' user". In some examples, the meaning is the same whether a tag appears before or after the base name, and the overall order of the tags does not matter.

[0224] A tagged entity refers to a subset of the entity class identified by a base name, namely those entities that have attributes matching each of the listed tags. Thus, "authorized, 'high priority', 'big shot' users" refers exactly to those users that have attributes matching all of the following tags: authorized, 'high priority', and 'big shot'. Many tag namespaces can be designated as ZPR tag namespaces. To reference a tag in a non-default tag namespace (ORG-ZPR), a user prefixes the tag namespace name with a prefix and a period. For example, if the tag namespace "myAuthTags" is designated as a ZPR tag namespace, and it has a tag called clearance, the tag would be referenced as myAuthtags.clearance. For example, "allow myAuthtags.clearance:secret users to read myAuthtags.clearance buckets". In some configurations, tags can be referenced by their key, with or without a value. For example, for the tag key "colors", to reference a specific value of the colors key, the tag can be referenced as "colors:red". According to some configurations, multiple tags can be referenced and logically ANDed together by a comma or space as a delimiter. Tags can also be conjoined using a with clause, where the tags are located before the object. For example, "tagl, tag2 users with tag3" means users with all three tags tagl, tag2, and tag3. As another example, "authorized 'high priority', 'big shot' users" refers to those users that have attributes matching all of the tags authorized, 'high priority', and 'big shot'.

[0225] A base name 830 refers to a simple name that can have associated tags (e.g., "words"). According to some configurations, base names 830 can include, but are not limited to, users 830A (e.g., "authorized users"), hosts 830B (e.g., "blue hosts"), the internet 830C (e.g., "all internet networks"), extranets 830D, resource categories 830E (e.g., endpoints, "Customer-app-2' endpoints"), and data 830F (e.g., "PII data").

[0226] According to some examples, verbs 840 include, but are not limited to, send 840A, receive 840B, send-receive 840C, and access 840D. Send 840A allows for initiating a connection (in the sense of stateful networking). Attributes 850 can be associated with different clauses. In some examples, users and data (target resource categories and endpoints) support attributes 850. Attributes 850 can be referenced in with clause 850A, without clause 1450B, and where clause 1450C. Attributes 850 are referenced in with clause 850A by using the attribute name followed by an equal sign and a string (e.g., name ='mygroup'). Multiple values that logically operate with an "or" operation can be referenced by using the keyword in, e.g., name in ('mygroup', 'yourgroup'). Some network-specific attribute examples include CIDR (v4 and v6), example: cidr = '10.10.10.10 / 12', IP address (v4 and v6), example: ip = '140.160.240.12', port, example port in (443, 557). And protocol, example protocol = 'ICMP'.

[0227] Different attributes can be associated with different resources. For example, example object storage bucket resource attributes can include, but are not limited to, approxmiateCount (int), number of objects, approximatesize (int), autoTieringString, isReadOnly, publicAccessType (string): NoPublicAccess, ObjectRead, ObjectReadWithoutList, StorageTier (String): Standard, Archive, and Versioning: String (Enabled, Suspended, Disabled). Example compute instance attributes include, but are not limited to, agentConfig, availabilityConfig, availabilityDomain, capacityReservationld, compartmentID, dedicatedMHostID, displayName, extendedMetadata (object), faultDomain, imageID, instanceOptions (object: areLegacyImdsEndpointsDisabled, (Boolean)), Opc-request-id, ipxeScript, launchMode, launchOptions (object) (bootValueType, firmware, isConsistentVolumeNamingEnabled, isPvEncryptionInTransitEnabled, networkType, remoteDataVolueType), lifecycleState (Moving, Provising, Running, Stopped, etc.), metadata, platformConfig, preemptibleInstanceconifg, region, shape, shapeConfig (object), sourceDetails (object), timeCreated, and timeMaintenanceRebootDue.

[0228] using name='value' or name in ( <set>Agreed upon attributes. Note that the value is a string literal and appears within single quotes. For example, attribute = 'value' or attribute in ('value1', 'value2'), which is an OR condition.

[0229] The with clause means those applicable to <base-name>attributes. The without clause has the opposite effect, where those attributes must not be present <base-name>Some examples include "Deny developer users access to dev servers without status='reserved'", "Deny developers without k:v", "Deny developers without key".

[0230] Scope 1460 identifies the scope of the policy declaration. The scope can include In identifier 860A (e.g., "in cubicle cl"), cubicle identifier 860B (e.g., "in cubicle cl"), Of identifier 860D (e.g., "of lease <lease name>", which can be used for admission / endorsement in cross-lease situations), and lease identifier 860E (e.g., "in lease cl"). Some examples include "Allow <tag> users to manage <tag> buckets in all cubicles cl", "Allow <tag> users to manage <tag> buckets in <tag> lease", "Allow <tag> users to manage <tag> buckets in <tag> cubicle cl", and "Allow <tag> users to manage <tag> buckets in <tag> cubicle cl:cl.l:cl.l.l".

[0231] Over network clause 870 within ZPL allows users to specify network locations that can be used for data packet flow. In some examples, a network location is a customer-controlled cloud infrastructure resource that represents a set of network devices. Over Network clause 870 allows users to distinguish their internet traffic from internal cloud infrastructure traffic and their off-net traffic, and to be able to require traffic to go through an SGW (rather than through the internet). These network devices can include an entire VCN, an enterprise network, or they can include a specific set of gateways (IGW, SGW, etc.) on a particular VCN. They are used in the over network clause to represent a set of network devices. From the perspective of ZPL, a network location simply represents a set of network resources. For example, "Allow any-zpr-tag users to manage colors:blue buckets in cubicle cl through colors:blue network" allows any user on any network participant to manage blue buckets, but only when the traffic flows on VCNs and gateways located in a network location with the blue tag.

[0232] Based on some examples, a resource-kind endpoint can be used. An endpoint represents the target host endpoint (e.g., IP address and port), and by default, access to all protocols / ports is allowed if not explicitly specified. However, if the resource-kind is the internet, then if the port / protocol is not specified in the declaration, an error may be generated and the policy cannot be saved because opening all protocols to the internet is not a secure default setting. A customer could write a policy like this: Allow the 'Customer-app-1' host to send and receive from the 'Customer-app-2' endpoint. This allows 'Customer-app-1' to communicate with 'Customer-app-2' via any protocol. If they want to be more specific, they could write a policy like this: Allow the 'Customer-app-1' host to send and receive from the 'Customer-app-2' endpoint using protocols in ('HTTP', 'SQL'). This further restricts communication to only HTTP and SQL protocols.

[0233] The advantage of the strategy described in this article is that it is data-centric. Users can write high-level data-centric policies that protect data from low-level (NSG, FW, networking) misconfigurations. Users do not need to keep L4 and L7 policies synchronized. Using ZPL, users can define policies to evaluate host tags (L4), evaluate gateway tags (e.g., network clauses), evaluate port / protocol protection for egress / ingress (L4), protect data from being leaked to the Internet (L4), protect data from being leaked to other leases (via multi-tenant endpoints) (L7), write unified data focus claims covering L4 and L7 authentication (L7), evaluate target tags (e.g., databases, buckets) (L7), evaluate user group tags (L7), evaluate resource-kinds (e.g., databases, buckets), enable data keywords (L7), evaluate OCI L7 permissions (inspect, read, use, manage) (L7), evaluate other context variables (name, etc.) (L7), evaluate locations (i.e., compartments) and other scopes (L7), evaluate L7 attributes (paths, cookies, etc.) (L7), and egress protection based on user / tag targets (L7).

[0234] Figure 9 The illustration depicts an example method 900 for performing packet-level, data-centric protection, based on various aspects. Method 900 can be... Figures 1-8 and Figures 14-18 one or more components. A computer-readable storage medium includes computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method 900. The method 900 can be performed in any suitable order. It should be recognized that the method 900 can include more or fewer steps than those depicted. Figure 9

[0235] At 902, a policy is created. As discussed above, the policy can include assertions / policy statements that use simple language to indicate security intent associated with data. For example, the policy statements can simply be "allow'red' data to travel within data zone 1," "prevent blue data from leaving lease X," "red data never leaves my lease," "blue data never reaches the internet," "blue data is not stored with red data," "green data never leaves data zone 2," and so forth.

[0236] At 904, a graph 614 is generated. As discussed above, a graph 614 can be generated that represents the intent of the security policy 616. In some examples, the rules engine 606 generates the graph 614 that includes possible policy or assertion statement results / outcomes for connections / neighbors to which data can potentially be passed. In some examples, the graph 614 describes possible data movement between resources modeled as vertices through edges of the graph 614. In some cases, the edges represent policy or assertion outcomes. To create the graph 614, the results / outcomes of applying the policy statements in the policy 616 are determined for different connections between the EPs 610 (e.g., neighbors of the enforcement points). For example, by referencing the nodes in the graph 614 (e.g., enforcement points in a network), it can be easily determined which other nodes (e.g., other enforcement points) the node is allowed to communicate with.

[0237] At 906, enforcement data 612 is generated. As discussed above, the rules engine 606 associated with the ZTA service 602 can access the policies in the policy 616 and generate enforcement data 612 that can be used by the EPs 610 for enforcement by analyzing the policies. The enforcement data is essentially a predefined instruction for each EP that specifies how the EP should handle data requests from any neighboring nodes based on a set of pre-evaluated policies associated with its origin ID. Similarly, the enforcement data specifies how to handle the transmission of outgoing data packets to any neighboring enforcement points or service endpoints defined in the graph based on pre-evaluated assertions associated with its origin ID. Figure 10 Additional information is provided about the origin ID.

[0238] ​At 908, the enforcement data is distributed to different enforcement points 610. As discussed above, after determining the enforcement data 612, the distribution engine 604 can provide the enforcement data 612 and / or make it available to different enforcement points 610 (e.g., smartNICs, gateways...) within one or more networks.

[0239] At 910, the enforcement points 610 enforce the rules identified by the enforcement data 612. As discussed above, the enforcement data 612 determined from the policy is enforced at the enforcement points 610 within one or more networks. As data packets are routed through the network, each enforcement point 610 can check the source and destination origin IDs, reference the enforcement data 612, and based on the policy either block the data packet or send the data packet to the next hop.

[0240] Figure 10 FIGURE illustrates an example method 1000 for preparing resources to enforce a policy, in accordance with various aspects. The method 1000 can be performed by one or more components of Figures 1-8 and Figures 14-18 A computer-readable storage medium comprises computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method 1000. The method 1000 can be performed in any suitable order. It should be appreciated that the method 1000 can include more or fewer steps than those depicted in Figure 10 .

[0241] At 1002, origin IDs are assigned to resources within one or more networks. In some configurations, a data security control plane (DSCP) assigns origin IDs to each resource and / or each enforcement point within one or more networks. As discussed above, EPs can include VNICs (e.g., smartNICs), gateways 630, and other resources involved in the transmission of data packets in one or more networks.

[0242] At 1004, resources can be tagged. As discussed above, data and other resources are tagged. Generally, resources can be automatically tagged using data discovery and / or manually tagged by a user. In some examples, a user tags resources such as, but not limited to, client applications (compute instances), IAM principals (groups and dynamic groups), networking gateways, data stored in storage services (e.g., databases, object storage buckets...), and the like.

[0243] In some configurations, a user tags a resource 1004, such as but not limited to tagging data 1004A stored in a storage service (e.g., database, object bucket...), tagging a host 1004B, tagging a gateway 1004C, tagging a smartNIC 1004D, tagging other enforcement points 610 (not shown), tagging an application 1004E (e.g., compute instance), tagging an IAM principal (group and dynamic group), tagging a client-defined resource 1004, and the like. As discussed above, a resource can have one or more associated tags, which include zero or more attributes.

[0244] At 1006, the tags are propagated to network resources. As discussed above, after (e.g., automatically / manually) tagging a resource, the tags are propagated by the service down to the network resources associated with it. For example, a customer can tag a compute instance with data classification tags, and the ZTA service 602 and / or some other device / component propagates these tags to the attached EPs (e.g., VNICs). In some examples, the service can expose endpoints. In these cases, data classification tags on the service resource can cause the VNICs to inherit the data classification tags, such that L4 network policies implemented at the VNICs cause proper enforcement of ZPR policies. Higher-level services use one of the virtual networking CP APIs to assign tags to the VNICs they own.

[0245] Figure 11 FIG. illustrates an example method 1100 for performing packet-level data-centric protection enforcement, in accordance with various aspects. The method 1100 can be performed by one or more components of Figures 1-8 and Figures 14-18 A computer-readable storage medium comprises computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method 1100. The method 1100 can be performed in any suitable order. It should be appreciated that the method 1100 can include more or fewer steps than those depicted in Figure 11

[0246] At 1102, a request to transfer data from a source resource (also referred to as a data source or data asset) that stores or has access to particular data to a target resource is received. In some examples, the source resource can be identified by an origin ID (e.g., a source origin ID) associated with the source resource, and the target resource can be identified by an origin ID (e.g., a target origin ID) associated with the target resource. In some examples, the request can be in the form of a data packet, where a header of the data packet includes a plurality of fields, including a field storing the origin ID of the source resource and another field storing the origin ID of the target resource.

[0247] ​The source resource can be a database, a host machine, a compute instance, etc. The target resource can be another database, a host machine, another compute instance, etc. In some examples, the source resource and the target source can be in a cloud infrastructure provided by a cloud service provider (CSP). In other examples, the source resource and the target source can be in different clouds provided by the same or different CSPs. For example, the source resource can be in a first cloud provided by a first CSP, while the target source can be in a second cloud provided by a different CSP. In still other use cases, the source resource and / or the target resource can be in an on-premises environment.

[0248] At 1104, the provenance IDs of the source resource and the target resource are determined. In some examples, the provenance IDs of the source resource and the target resource can be extracted from the request packet, and the source resource and the target resource are identified based on the extracted provenance IDs.

[0249] At 1106, a determination is made as to whether the transfer is allowed. In some examples, a signal can be sent to an evaluator component configured to determine whether the particular data is permitted to be transferred from the source resource to the target resource. The evaluator can be associated with the source resource or the target resource. For example, if the source resource is a compute instance, then the evaluator can be a smartNIC (or generally a network virtualization device (NVD)) such as a virtual network interface card (VNIC) that implements that source compute instance.

[0250] At 1108, the zone membership of the source resource and the target resource is identified. For example, information can be obtained from the rules engine 606 to determine that the source resource belongs to a “source zone” and the target resource belongs to a “target zone.” In other examples, information can be obtained from the rules engine 606 to determine that both the source resource and the target resource belong to the same zone.

[0251] At 1110, assertions associated with the source zone are identified and evaluated for the target resource. According to some examples, the evaluation of the assertions can take into account attributes associated with the source resource and the target resource, as well as security-related attributes configured for the particular data.

[0252] At 1112, a determination is made as to whether the assertions are violated. If any of the assertions are not satisfied (or are violated), then the process moves to 1114. If the assertions are not violated, then the process moves to 1116.

[0253] At 1114, the target resource is denied access to the particular data, and the process can return to perform other actions. In certain implementations, an error message can be communicated to the target resource indicating that the particular data is not permitted to be transferred to the target resource.

[0254] At 1116, the policies defined between the source region and the target region are identified and evaluated. In some examples, the evaluation of the policies can take into account attributes associated with the source resource and the target resource, as well as security-related attributes configured for the particular data. The evaluation can be performed using predetermined access decisions made based on the attributes associated with the source resource and the target resource, as well as the security-related attributes configured for the particular data.

[0255] At 1118, a determination is made as to whether the policies are violated. If any of the policies are not satisfied (or are violated), then the process moves to 1120. If the policies are not violated, then the process moves to 1122.

[0256] At 1120, the target resource is denied access to the particular data, and the process can return to perform other actions. In certain implementations, an error message can be communicated to the target resource indicating that the particular data is not permitted to be communicated to the target resource.

[0257] At 1122, the particular data is permitted to be communicated from the source resource to the target resource. In this way, if all of the relevant assertions and policies are successfully verified (i.e., not violated), then the particular data will be allowed to be communicated from the source resource to the target resource.

[0258] Figure 12 FIG. illustrates an example method 1200 of performing packet-level data-centric protection enforcement at a sending EP 610, in accordance with various aspects. The method 1900 can be performed by one or more components of Figures 1-8 and Figures 14-18 A computer-readable storage medium comprises computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method 1200. The method 1200 can be performed in any suitable order. It should be recognized that the method 1200 can include more or fewer steps than those depicted in Figure 12

[0259] At 1202, the identity of the sender is checked, as well as the policies to be enforced. In some examples, the EP 610 involved in the transmission of the packet can determine whether the packet should be sent to the next hop / destination or blocked from being sent to the next hop / destination, rather than relying on a receiving device to enforce network policies.

[0260] At 1204, the rules to be enforced are determined. As discussed above, the enforcement data 612 can be received from the ZTA service 602, the distribution engine 604, and / or from some other device component. As discussed above, the enforcement data 612 is associated with controlling the flow of traffic through one or more networks.

[0261] ​At 1206, a determination is made as to whether the EP 610 is authorized to transmit the data packet to the next hop / destination. As discussed above, if the EP 610 determines that each rule that is enforced passes the specified condition based on the enforcement data 612, then the process flows to 1208. When one or more of the rules do not pass the specified condition, the process flows to 1210.

[0262] At 1208, the data packet is transmitted. As discussed above, in some examples the EP 610 transmits the data packet when authorized based on the policy.

[0263] At 1210, an alert can be triggered when a determination is made. In some examples, an alert mode can be used to determine which data packets will be allowed / denied without actually preventing the data packets from being transmitted / received.

[0264] Figure 13 FIG. illustrates an example method 1300 of performing data packet level data-centric protection enforcement at a receiving EP 610, in accordance with various aspects. The method 1300 can be performed by one or more components of Figures 1-8 and Figures 14-18 A computer-readable storage medium comprises computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method 1300. The method 1300 can be performed in any suitable order. It should be recognized that the method 1300 can include more or fewer steps than those depicted in Figure 13 .

[0265] At 1302, a data packet is received. As discussed above, the data packet can be received from another EP 610 in a network flow between a source and a destination.

[0266] At 1304, the identity of the sender, the recipient, and the policy to be enforced are checked. As discussed above, any / all EP 610 involved in the transmission of the data packet can determine whether the data packet should be sent to the next hop / destination or prevented from being sent to the next hop / destination, rather than relying solely on the receiving device to enforce the network policy.

[0267] At 1306, enforcement data 612 associated with the receiving EP 610 is accessed. As discussed above, the enforcement data 612 can be received from the ZTA service 602, the distribution engine 604, and / or from some other device component. As discussed above, the enforcement data 612 is associated with controlling the flow of traffic through one or more networks.

[0268] At 1308, a determination is made as to whether the EP 610 is authorized to receive and / or transmit the data packet to the next hop / destination. As discussed above, if the EP 610 determines that each of the rules that are enforced pass the specified condition based on the enforcement data 612, then the process flows to 1310. When one or more of the rules do not pass the specified condition, the process flows to 1312.

[0269] At 1310, the data packet is received / transmitted. As discussed above, in some examples, the EP 610 receives the data packet (e.g., for processing) and / or transmits the data packet when authorized based on the policy.

[0270] At 1312, an alert can be triggered when determined. In some examples, an alert mode can be used to determine which data packets will be allowed / denied without actually blocking the data packets from being transmitted / received.

[0271] Example CSPI architecture for providing cloud services

[0272] As noted above, infrastructure as a service (IaaS) is a particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provision various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Thus, because these services can be policy driven, an IaaS user can be able to implement policies to drive load balancing to maintain availability and performance of applications.

[0273] In some cases, an IaaS customer can access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider’s services to install the remaining elements of an application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider’s services to perform various functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0274] In most cases, the cloud computing model will require the involvement of a cloud provider. The cloud provider can but does not necessarily have to be a third party service that specializes in providing (e.g., provisioning, renting, selling) IaaS. An entity can also choose to deploy a private cloud, thereby becoming its own infrastructure service provider.

[0275] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It can also include the process of preparing a server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer can be responsible for the process of (OS), middleware, and / or application deployment (e.g., onto a self-service virtual machine, etc. (which can be launched on-demand, for example)).

[0276] In some examples, IaaS provisioning can refer to the acquisition of computers or virtual hosts for use, even with the installation of required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning can need to be performed first.

[0277] In some cases, there are two different challenges to IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is running. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.) once everything has been provisioned. In some cases, both challenges can be addressed by enabling a configuration that defines the infrastructure in a declarative manner. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, a workflow can be generated that creates and / or manages the different components described in the configuration file.

[0278] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., a potential on-demand pool of configurable and / or shareable computing resources), also referred to as a core network. In some examples, one or more ingress / egress traffic group rules can also be provisioned to define how to set up the ingress / egress traffic of the network as well as one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. can also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve step by step.

[0279] In some cases, continuous deployment techniques can be employed to enable deployment of infrastructure code across a variety of virtual computing environments. Further, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is expected to be deployed to one or more, but typically many, different production environments (e.g., across a variety of different geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned with a provisioning tool, and / or code can be deployed with a deployment tool once the infrastructure is provisioned.

[0280] Figure 14 is a block diagram 1400 illustrating an example pattern of an IaaS architecture, in accordance with at least one embodiment. A service operator 1402 can be communicatively coupled to a secure host tenancy 1404 that can include a virtual cloud network (VCN) 1406 and a secure host subnet 1408. In some examples, the service operator 1402 can use one or more client computing devices, which can be portable handheld devices (e.g., iPhone®, cellular phones, iPad®, computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google Glass® head mounted display), running software such as Microsoft Windows Mobile® and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting Internet, e-mail, Short Message Service (SMS), Blackberry®, or other communications protocols. Alternatively, the client computing devices can be general purpose personal computers including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing devices can be workstation computers running any of a variety of commercially-available UNIX® or UNIX-like operating systems, including, without limitation, the variety of GNU / Linux operating systems (which are based on the Linux kernel), such as, for example, Google Chrome OS. Alternatively or additionally, client computing devices can be any other electronic device capable of

[0281] VCN 1406 can include a local peering gateway (LPG) 1410 that can be communicatively coupled to a secure shell (SSH) VCN 1412 via the LPG 1410 contained in the SSH VCN 1412. The SSH VCN 1412 can include an SSH subnet 1414, and the SSH VCN 1412 can be communicatively coupled to a control plane VCN 1416 via the LPG 1410 contained in the control plane VCN 1416. Further, the SSH VCN 1412 can be communicatively coupled to a data plane VCN 1418 via the LPG 1410. The control plane VCN 1416 and the data plane VCN 1418 can be contained in a service tenancy 1419 that can be owned and / or operated by an IaaS provider.

[0282] The control plane VCN 1416 can include a control plane demilitarized zone (DMZ) tier 1420 that acts as a perimeter network (e.g., a portion of a company network between a company intranet and an external network). DMZ-based servers can assume limited responsibilities and help control vulnerabilities. Further, the DMZ tier 1420 can include one or more load balancer (LB) subnets 1422, a control plane application tier 1424 that can include application subnet(s) 1426, a control plane data tier 1428 that can include database (DB) subnet(s) 1430 (e.g., front-end DB subnet(s) and / or back-end DB subnet(s)). The LB subnet(s) 1422 contained in the control plane DMZ tier 1420 can be communicatively coupled to the application subnet(s) 1426 contained in the control plane application tier 1424 and to an internet gateway 1434 that can be contained in the control plane VCN 1416, and the application subnet(s) 1426 can be communicatively coupled to the DB subnet(s) 1430 contained in the control plane data tier 1428 as well as to a service gateway 1436 and a network address translation (NAT) gateway 1438. The control plane VCN 1416 can include the service gateway 1436 and the NAT gateway 1438.

[0283] The control plane VCN 1416 can include a data plane mirror application tier 1440 that can include the application subnet(s) 1426. The application subnet(s) 1426 contained in the data plane mirror application tier 1440 can include a virtual network interface controller (VNIC) 1442 that can execute a compute instance 1444. The compute instance 1444 can communicatively couple the application subnet(s) 1426 of the data plane mirror application tier 1440 to the application subnet(s) 1426 that can be contained in a data plane application tier 1446.

[0284] The data plane VCN 1418 can include a data plane app tier 1446, a data plane DMZ tier 1448, and a data plane data tier 1450. The data plane DMZ tier 1448 can include LB subnet(s) 1422, which can be communicatively coupled to app subnet(s) 1426 of the data plane app tier 1446 and internet gateway 1434 of the data plane VCN 1418. The app subnet(s) 1426 can be communicatively coupled to service gateway 1436 of the data plane VCN 1418 and NAT gateway 1438 of the data plane VCN 1418. The data plane data tier 1450 can also include DB subnet(s) 1430, which can be communicatively coupled to the app subnet(s) 1426 of the data plane app tier 1446.

[0285] The internet gateway 1434 of the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to a metadata management service 1452, which can be communicatively coupled to the public internet 1454. The public internet 1454 can be communicatively coupled to the NAT gateway 1438 of the control plane VCN 1416 and the data plane VCN 1418. The service gateway 1436 of the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to cloud services 1456.

[0286] In some examples, the service gateway 1436 of the control plane VCN 1416 or the data plane VCN 1418 can make application programming interface (API) calls to the cloud services 1456 without going through the public internet 1454. The API calls from the service gateway 1436 to the cloud services 1456 can be one-way: the service gateway 1436 can make API calls to the cloud services 1456, and the cloud services 1456 can send requested data to the service gateway 1436. However, the cloud services 1456 can not initiate API calls to the service gateway 1436.

[0287] In some examples, the secure host tenancy 1404 can be directly connected to the service tenancy 1419, which can otherwise be isolated. The secure host subnet 1408 can communicate with the SSH subnet 1414 through the LPG 1410, which can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1408 to the SSH subnet 1414 can enable the secure host subnet 1408 to access other entities within the service tenancy 1419.

[0288] The control plane VCN 1416 can allow users of the service tenancy 1419 to set up or otherwise provision desired resources. Desired resources provisioned in the control plane VCN 1416 can be deployed or otherwise used in the data plane VCN 1418. In some examples, the control plane VCN 1416 can be isolated from the data plane VCN 1418, and a data plane mirror application tier 1440 of the control plane VCN 1416 can communicate with a data plane application tier 1446 of the data plane VCN 1418 via a VNIC 1442, which can be contained in the data plane mirror application tier 1440 and the data plane application tier 1446.

[0289] In some examples, a user or customer of the system can make a request, such as a create, read, update, or delete (CRUD) operation, through the public Internet 1454, which can communicate the request to the metadata management service 1452. The metadata management service 1452 can communicate the request to the control plane VCN 1416 through the Internet gateway 1434. The request can be received by the LB subnet(s) 1422 contained in the control plane DMZ tier 1420. The LB subnet(s) 1422 can determine that the request is valid, and in response to the determination, the LB subnet(s) 1422 can transmit the request to the application subnet(s) 1426 contained in the control plane application tier 1424. If the request is validated and requires a call to the public Internet 1454, the call to the public Internet 1454 can be transmitted to the NAT gateway 1438, which can make the call to the public Internet 1454. Metadata that can be desired to be stored can be stored in the DB subnet(s) 1430.

[0290] In some examples, the data plane mirror application tier 1440 can facilitate direct communication between the control plane VCN 1416 and the data plane VCN 1418. For example, it can be desired to apply changes, updates, or other appropriate modifications to configuration to resources contained in the data plane VCN 1418. Via the VNIC 1442, the control plane VCN 1416 can directly communicate with the resources contained in the data plane VCN 1418, and thereby can perform the changes, updates, or other appropriate modifications to configuration.

[0291] In some embodiments, the control plane VCN 1416 and the data plane VCN 1418 can be contained in a service tenancy 1419. In this case, a user or customer of the system can not own or operate the control plane VCN 1416 or the data plane VCN 1418. Instead, the IaaS provider can own or operate the control plane VCN 1416 and the data plane VCN 1418, both of which can be contained in the service tenancy 1419. This embodiment can enable isolation of networks that can prevent a user or customer from interacting with resources of other users or other customers. Further, this embodiment can allow a user or customer of the system to privately store databases without having to rely on the public Internet 1454, which can not have the desired level of threat protection.

[0292] In other embodiments, the LB subnet(s) 1422 contained in the control plane VCN 1416 can be configured to receive signals from the service gateway 1436. In this embodiment, the control plane VCN 1416 and the data plane VCN 1418 can be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1454. This embodiment can be desirable for a customer of the IaaS provider because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on the service tenancy 1419, which can be isolated from the public Internet 1454.

[0293] Figure 15 is a block diagram 1500 illustrating another example pattern of an IaaS architecture, in accordance with at least one embodiment. A service operator 1502 (e.g., the service operator 1402 of Figure 14 ) can be communicatively coupled to a secure host tenancy 1504 (e.g., the secure host tenancy 1404 of Figure 14 ). The secure host tenancy 1504 can include a virtual cloud network (VCN) 1506 (e.g., the VCN 1406 of Figure 14 ) and a secure host subnet 1508 (e.g., the secure host subnet 1408 of Figure 14 ). The VCN 1506 can include a local peering gateway (LPG) 1510 (e.g., the LPG 1410 of Figure 14 ) that can be communicatively coupled to a secure shell (SSH) VCN 1512 (e.g., the SSH VCN 1412 of Figure 14 ) via the LPG 1410 contained in the SSH VCN 1512. The SSH VCN 1512 can include an SSH subnet 1514 (e.g., the SSH subnet 1404 of Figure 14 SSH subnet 1414), and SSH VCN 1512 can be communicatively coupled to control plane VCN 1516 via LPG 1510 included in control plane VCN 1516 (e.g., Figure 14 Control plane VCN 1416). Control plane VCN 1516 may be included in service lease 1519 (e.g., Figure 14 In the service lease 1419), and the data plane VCN 1518 (e.g., Figure 14 The data plane VCN 1418 may be included in a customer lease 1521 that may be owned or operated by the system’s users or customers.

[0294] The control plane VCN 1516 may include one or more LB subnets 1522 (e.g., Figure 14 The control plane DMZ layer 1520 of (one or more) LB subnets 1422) (e.g., Figure 14 The control plane DMZ layer 1420 may contain one or more application subnets 1526 (e.g., Figure 14 The control plane application layer 1524 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424 can contain one or more database (DB) subnets 1530 (e.g., similar to...). Figure 14 The control plane data layer 1528 of (one or more) DB subnets 1430 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1522 contained in the control plane DMZ layer 1520 can be communicatively coupled to one or more application subnets 1526 contained in the control plane application layer 1524 and an Internet gateway 1534 that can be contained in the control plane VCN 1516 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1526 can communicatively couple to DB subnet(s) 1530 contained in control plane data layer 1528 and service gateway 1536 (e.g., Figure 14 Service gateway 1436) and Network Address Translation (NAT) gateway 1538 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0295] The control plane VCN 1516 may include a data plane mirror application layer 1540 that may contain one or more application subnets 1526 (e.g., Figure 14 application layer 1440). The application subnet(s) 1526 contained in the data plane mirror application layer 1540 can include virtual network interface controllers (VNICs) 1542 that can execute compute instances 1544 (e.g., similar to the compute instances 1444 of FIG. 14). The compute instances 1544 can facilitate communication between the application subnet(s) 1526 of the data plane mirror application layer 1540 and the application subnet(s) 1526 that can be contained in a data plane application layer 1546 (e.g., the data plane application layer 1446 of FIG. 14) via the VNICs 1542 contained in the data plane mirror application layer 1540 and the VNICs 1542 contained in the data plane application layer 1546. Figure 14 Figure 14 application layer 1446) via the VNICs 1542 contained in the data plane mirror application layer 1540 and the VNICs 1542 contained in the data plane application layer 1546.

[0296] The internet gateway 1534 contained in the control plane VCN 1516 can be communicatively coupled to a metadata management service 1552 (e.g., the metadata management service 1452 of FIG. 14) that can be communicatively coupled to a public internet 1554 (e.g., the public internet 1454 of FIG. 14). The public internet 1554 can be communicatively coupled to the NAT gateway 1538 contained in the control plane VCN 1516. The service gateway 1536 contained in the control plane VCN 1516 can be communicatively coupled to a cloud service 1556 (e.g., the cloud service 1456 of FIG. 14). Figure 14 Figure 14 Figure 14

[0297] In some examples, the data plane VCN 1518 can be contained in a customer tenancy 1521. In this case, the IaaS provider can provide a control plane VCN 1516 for each customer, and the IaaS provider can set up a unique compute instance 1544 contained in the service tenancy 1519 for each customer. Each compute instance 1544 can allow for communication between the control plane VCN 1516 contained in the service tenancy 1519 and the data plane VCN 1518 contained in the customer tenancy 1521. The compute instance 1544 can allow resources provisioned in the control plane VCN 1516 contained in the service tenancy 1519 to be deployed or otherwise used in the data plane VCN 1518 contained in the customer tenancy 1521.

[0298] ​​​​In other examples, a customer of the IaaS provider can have a database that exists in the customer tenancy 1521. In this example, the control plane VCN 1516 can include a data plane mirror application tier 1540, which can include application subnet(s) 1526. The data plane mirror application tier 1540 can reside in the data plane VCN 1518, but the data plane mirror application tier 1540 can not be in the data plane VCN 1518. That is, the data plane mirror application tier 1540 can access the customer tenancy 1521, but the data plane mirror application tier 1540 can not exist in the data plane VCN 1518 or be owned or operated by the customer of the IaaS provider. The data plane mirror application tier 1540 can be configured to make calls to the data plane VCN 1518, but can not be configured to make calls to any entities contained in the control plane VCN 1516. The customer can desire to deploy or otherwise use resources provisioned in the control plane VCN 1516 in the data plane VCN 1518, and the data plane mirror application tier 1540 can facilitate the customer’s desired deployment or other use of the resources.

[0299] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN 1518. In this embodiment, the customer can determine what the data plane VCN 1518 can access, and the customer can limit access to the public Internet 1554 from the data plane VCN 1518. The IaaS provider can not be able to apply filters or otherwise control access to any external networks or databases from the data plane VCN 1518. The customer applying filters and controls to the data plane VCN 1518 contained in the customer tenancy 1521 can help isolate the data plane VCN 1518 from other customers and the public Internet 1554.

[0300] In some embodiments, cloud services 1556 can be invoked by service gateways 1536 to access services that can not be present on public internet 1554, control plane VCN 1516, or data plane VCN 1518. Connections between cloud services 1556 and control plane VCN 1516 or data plane VCN 1518 can not be real-time or continuous. Cloud services 1556 can exist on a different network owned or operated by the laaS provider. Cloud services 1556 can be configured to receive invocations from service gateways 1536 and can be configured to not receive invocations from public internet 1554. Some cloud services 1556 can be isolated from other cloud services 1556, and control plane VCN 1516 can be isolated from cloud services 1556 that can not be in the same region as control plane VCN 1516. For example, control plane VCN 1516 can be located in "Region 1" and cloud service "Deployment 11" can be located in Region 1 and "Region 2." If a service gateway 1536 contained in control plane VCN 1516 located in Region 1 makes an invocation to Deployment 11, the invocation can be transmitted to Deployment 11 in Region 1. In this example, control plane VCN 1516 or Deployment 11 in Region 1 can not be communicatively coupled or otherwise in communication with Deployment 11 in Region 2.

[0301] Figure 16 is a block diagram 1600 illustrating another example pattern of an laaS architecture, in accordance with at least one embodiment. A service operator 1602 (e.g., service operator 1402 of Figure 14 ) can be communicatively coupled to a secure host tenancy 1604 (e.g., secure host tenancy 1404 of Figure 14 ). Secure host tenancy 1604 can include a virtual cloud network (VCN) 1606 (e.g., VCN 1406 of Figure 14 ) and a secure host subnet 1608 (e.g., secure host subnet 1408 of Figure 14 ). VCN 1606 can include an LPG 1610 (e.g., LPG 1410 of Figure 14 ) that can be communicatively coupled to an SSH VCN 1612 (e.g., SSH VCN 1412 of Figure 14 ) via LPG 1610 contained in SSH VCN 1612. SSH VCN 1612 can include an SSH subnet 1614 (e.g., SSH subnet 1414 of Figure 14 ) and SSH VCN 1612 can be communicatively coupled to a control plane VCN 1616 (e.g., control plane VCN 1416 of Figure 14 The control plane VCN 1416) and coupled to the data plane VCN 1618 via the LPG 1610 contained in the data plane VCN 1618 (e.g., Figure 14 Data plane 1418). Control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619 (e.g., Figure 14 (Service rental 1419).

[0302] The control plane VCN 1616 may include a subnet 1622 that may contain one or more load balancer (LB) subnets (e.g., Figure 14 The control plane DMZ layer 1620 of (one or more) LB subnets 1422) (e.g., Figure 14 The control plane DMZ layer 1420 may include one or more application subnets 1626 (e.g., similar to...). Figure 14 The control plane application layer 1624 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424), and may include (one or more) DB subnets 1630, and the control plane data layer 1628 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 can be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that can be contained in the control plane VCN 1616 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1626 can communicatively couple to DB subnet(s) 1630 contained in control plane data layer 1628 and service gateway 1636 (e.g., Figure 14 The service gateway) and Network Address Translation (NAT) gateway 1638 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0303] The data plane VCN 1618 may include the data plane application layer 1646 (e.g., Figure 14 Data plane application layer 1446), data plane DMZ layer 1648 (e.g., Figure 14 Data plane DMZ layer 1448), and data plane data layer 1650 (e.g., Figure 14 Data Plane DMZ Tier 1648 can include Trusted Application Subnet(s) 1660 and Untrusted Application Subnet(s) 1662 that can be communicatively coupled to Data Plane Application Tier 1646 and LB Subnet(s) 1622 that include Internet Gateway 1634 in Data Plane VCN 1618. Trusted Application Subnet(s) 1660 can be communicatively coupled to Service Gateway 1636 included in Data Plane VCN 1618, NAT Gateway 1638 included in Data Plane VCN 1618, and DB Subnet(s) 1630 included in Data Plane Data Tier 1650. Untrusted Application Subnet(s) 1662 can be communicatively coupled to Service Gateway 1636 included in Data Plane VCN 1618 and DB Subnet(s) 1630 included in Data Plane Data Tier 1650. Data Plane Data Tier 1650 can include DB Subnet(s) 1630 that can be communicatively coupled to Service Gateway 1636 included in Data Plane VCN 1618.

[0304] Untrusted Application Subnet(s) 1662 can include one or more Primary VNICs 1664(1)-(N) that can be communicatively coupled to Tenant Virtual Machines (VMs) 1666(1)-(N). Each Tenant VM 1666(1)-(N) can be communicatively coupled to a respective Application Subnet 1667(1)-(N) that can be included in a respective Container egress VCN 1668(1)-(N) that can be included in a respective Customer tenancy 1670(1)-(N). A respective Secondary VNIC 1672(1)-(N) can facilitate communication between Untrusted Application Subnet(s) 1662 included in Data Plane VCN 1618 and Application Subnets included in Container egress VCNs 1668(1)-(N). Each Container egress VCN 1668(1)-(N) can include NAT Gateway 1638 that can be communicatively coupled to Public Internet 1654 (e.g., Figure 14 Public Internet 1454).

[0305] Internet Gateway 1634 included in Control Plane VCN 1616 and included in Data Plane VCN 1618 can be communicatively coupled to Meta Data Management Service 1652 (e.g., Figure 14 Metadata Management Service 1452), which can be communicatively coupled to the public Internet 1654. The public Internet 1654 can be communicatively coupled to a NAT gateway 1638 contained in the control plane VCN 1616 and contained in the data plane VCN 1618. A service gateway 1636 contained in the control plane VCN 1616 and contained in the data plane VCN 1618 can be communicatively coupled to cloud services 1656.

[0306] In some embodiments, the data plane VCN 1618 can be integrated with customer tenancies 1670. In some cases, such as in cases where support can be desired when executing code, such integration can be useful or desirable to customers of the IaaS provider. Customers can provide code that can be disruptive, can communicate with other customer resources, or can otherwise cause undesirable effects to run. In response to this, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0307] In some examples, a customer of the IaaS provider can grant temporary network access to the IaaS provider and request functionality to be attached to the data plane tier application 1646. Code to run the functionality can be executed in VMs 1666(1)-(N), and the code can not be configured to run anywhere else on the data plane VCN 1618. Each VM 1666(1)-(N) can be connected to one customer tenancy 1670. Respective containers 1671(1)-(N) contained in the VMs 1666(1)-(N) can be configured to run the code. In this case, there can be double isolation (e.g., the containers 1671(1)-(N) run the code, where the containers 1671(1)-(N) can be contained in at least the VMs 1666(1)-(N) contained in the untrusted app subnet(s) 1662), which can help prevent incorrect or otherwise undesirable code from damaging the network of the IaaS provider or damaging the network of a different customer. The containers 1671(1)-(N) can be communicatively coupled to the customer tenancy 1670 and can be configured to transmit or receive data from the customer tenancy 1670. The containers 1671(1)-(N) can not be configured to transmit or receive data from any other entity in the data plane VCN 1618. Upon completion of running the code, the IaaS provider can terminate or otherwise dispose of the containers 1671(1)-(N).

[0308] In some embodiments, one or more trusted application subnets 1660 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1660 may be communicatively coupled to one or more database subnets 1630 and configured to perform CRUD operations in one or more database subnets 1630. One or more untrusted application subnets 1662 may be communicatively coupled to one or more database subnets 1630, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1630. Containers 1671(1)-(N) that may be contained in each customer's VM 1666(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1630.

[0309] In other embodiments, the control plane VCN 1616 and the data plane VCN 1618 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1616 and the data plane VCN 1618. However, communication can occur indirectly through at least one method. The LPG 1610 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1616 and the data plane VCN 1618. In another example, the control plane VCN 1616 or the data plane VCN 1618 may invoke the cloud service 1656 via the service gateway 1636. For example, an invocation of the cloud service 1656 from the control plane VCN 1616 may include a request for a service that can communicate with the data plane VCN 1618.

[0310] Figure 17 This is a block diagram 1700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1702 (e.g., Figure 14 The service provider 1402 can communicatively couple to the secure host lease 1704 (e.g., Figure 14 Secure hosting lease 1404), the secure hosting lease 1704 may include a virtual cloud network (VCN) 1706 (e.g., Figure 14 VCN 1406) and Secure Host Subnet 1708 (e.g., Figure 14 The secure host subnet 1408). VCN 1706 can include LPG 1710 (e.g., Figure 14 The LPG 1410), the LPG 1710 can be accessed via SSH VCN 1712 (e.g., LPG 1410), Figure 14 The LPG 1710 in SSH VCN 1412 is communicatively coupled to SSH VCN 1712. SSH VCN 1712 may include SSH subnet 1714 (e.g., Figure 14 SSH subnet 1414), and SSH VCN 1712 can be communicatively coupled to control plane VCN 1716 via LPG 1710 contained in control plane VCN 1716 (e.g., Figure 14 The control plane VCN 1416) and coupled to the data plane VCN 1718 via the LPG 1710 contained in the data plane VCN 1718 (e.g., Figure 14 Data plane 1418). Control plane VCN 1716 and data plane VCN 1718 may be included in service lease 1719 (e.g., Figure 14 (Service rental 1419).

[0311] The control plane VCN 1716 may include one or more LB subnets 1722 (e.g., Figure 14 The control plane DMZ layer 1720 of (one or more) LB subnets 1422) (e.g., Figure 14 The control plane DMZ layer 1420 may include one or more application subnets 1726 (e.g., Figure 14 The control plane application layer 1724 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424 may include (one or more) DB subnets 1730 (e.g., Figure 16 The control plane data layer 1728 of (one or more) DB subnets 1630 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1722 contained in the control plane DMZ layer 1720 can be communicatively coupled to one or more application subnets 1726 contained in the control plane application layer 1724 and an Internet gateway 1734 that can be contained in the control plane VCN 1716 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1726 can communicatively couple to DB subnet(s) 1730 contained in control plane data layer 1728 and service gateway 1736 (e.g., Figure 14 The service gateway) and Network Address Translation (NAT) gateway 1738 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0312] Data plane VCN 1718 may include data plane application layer 1746 (e.g., Figure 14 Data plane application layer 1446), data plane DMZ layer 1748 (e.g., Figure 14 Data plane DMZ layer 1448), and data plane data layer 1750 (e.g., Figure 14 The data plane data layer 1450). The data plane DMZ layer 1748 may include one or more trusted application subnets 1760 that can be communicatively coupled to the data plane application layer 1746 (e.g., Figure 16 (one or more) trusted application subnets 1660) and (one or more) untrusted application subnets 1762 (e.g., Figure 16 The data plane VCN 1718 may include one or more untrusted application subnets 1762 and one or more LB subnets 1722. One or more trusted application subnets 1760 may be communicatively coupled to a service gateway 1736, a NAT gateway 1738, and a DB subnet 1730, all contained in the data plane VCN 1718. One or more untrusted application subnets 1762 may be communicatively coupled to a service gateway 1736 and a DB subnet 1730, all contained in the data plane VCN 1718 and the data plane data layer 1750, respectively. The data plane data layer 1750 may include one or more DB subnets 1730 that may be communicatively coupled to a service gateway 1736, all contained in the data plane VCN 1718.

[0313] One or more untrusted application subnets 1762 may include a primary VNIC 1764(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N) residing within one or more untrusted application subnets 1762. Each tenant VM 1766(1)-(N) may run code in a corresponding container 1767(1)-(N) and is communicatively coupled to an application subnet 1726 that may be contained in a data plane application layer 1746 contained in a container egress VCN 1768. A corresponding secondary VNIC 1772(1)-(N) may facilitate communication between one or more untrusted application subnets 1762 contained in a data plane VCN 1718 and the application subnets contained in a container egress VCN 1768. The container egress VCN may include a public internet 1754 (e.g., Figure 14 The public internet (1454) uses NAT gateway 1738.

[0314] The Internet gateway 1734 included in the control plane VCN 1716 and included in the data plane VCN 1718 can be communicatively coupled to a metadata management service 1752 (e.g., a metadata management system 1452 of Figure 14 FIG. 1), which can be communicatively coupled to the public Internet 1754. The public Internet 1754 can be communicatively coupled to the NAT gateway 1738 included in the control plane VCN 1716 and included in the data plane VCN 1718. The service gateway 1736 included in the control plane VCN 1716 and included in the data plane VCN 1718 can be communicatively coupled to the cloud services 1756.

[0315] In some examples, Figure 17 The mode shown in the architecture of the block diagram 1700 can be considered an exception to the mode shown in the architecture of the block diagram 1600 of Figure 16 FIG. 1) and can be desirable to customers of the IaaS provider if the IaaS provider cannot directly communicate with the customers (e.g., a disconnected region). Each customer’s VM 1766(1)-(N) can be configured to access in real-time a respective container 1767(1)-(N) included in the VM 1766(1)-(N). The container 1767(1)-(N) can be configured to make calls to a respective secondary VNIC 1772(1)-(N) included in the application subnet(s) 1726 of the data plane application tier 1746, which can be included in the container egress VCN 1768. The secondary VNIC 1772(1)-(N) can transmit the calls to the NAT gateway 1738, which can transmit the calls to the public Internet 1754. In this example, the containers 1767(1)-(N), which can be accessed in real-time by the customer, can be isolated from the control plane VCN 1716 and can be isolated from other entities included in the data plane VCN 1718. The containers 1767(1)-(N) can also be isolated from resources from other customers.

[0316] In other examples, a customer can use containers 1767(1)-(N) to invoke cloud services 1756. In this example, the customer can run code in containers 1767(1)-(N) that requests services from cloud services 1756. Containers 1767(1)-(N) can transmit the request to secondary VNICs 1772(1)-(N), which can transmit the request to a NAT gateway, which can transmit the request to public internet 1754. Public internet 1754 can transmit the request to LB subnet(s) 1722 contained in control plane VCN 1716 via internet gateway 1734. In response to determining that the request is valid, the LB subnet(s) can transmit the request to application subnet(s) 1726, which can transmit the request to cloud services 1756 via service gateway 1736.

[0317] It should be appreciated that the IaaS architectures 1400, 1500, 1600, 1700 depicted in the figures can have other components than those depicted. In addition, the embodiments shown in the figures are merely examples of cloud infrastructure systems with which embodiments of the present disclosure can be practiced. In some other embodiments, an IaaS system can have more or fewer components than shown in the figures, can combine two or more components, or can have a different configuration or arrangement of components.

[0318] In certain embodiments, the IaaS systems described herein can include an application suite, middleware, and database service products delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0319] Figure 18 An example computer system 1800 in which various embodiments can be implemented is illustrated in FIG. 18. System 1800 can be used to implement any of the computer systems described above. As shown in FIG. 18, computer system 1800 includes a processing unit 1804 that communicates with a number of peripheral subsystems via a bus subsystem 1802. These peripheral subsystems can include a processing acceleration unit 1806, an I / O subsystem 1808, a storage subsystem 1818, and a communication subsystem 1824. Storage subsystem 1818 includes tangible computer-readable storage media 1822 and a system memory 1810.

[0320] Bus subsystem 1802 provides a mechanism for letting the various components and subsystems of computer system 1800 communicate with each other as intended. Although bus subsystem 1802 is illustrated as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. Bus subsystem 1802 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as Mezzanine bus manufactured to the IEEE P1386.1 standard.

[0321] Processing unit 1804, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system 1800. One or more processors can be included in processing unit 1804. These processors can include single or multiple threaded processors. In certain embodiments, processing unit 1804 can be implemented as one or more independent processing units 1832 and / or 1834, with single or multiple threaded processors included in each processing unit. In other embodiments, processing unit 1804 can also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0322] In various embodiments, processing unit 1804 can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processor(s) 1804 and / or in storage subsystem 1818. Through suitable programming, processor(s) 1804 can provide various functionalities described above. Computer system 1800 can additionally include a processing acceleration unit 1806, which can include a digital signal processor (DSP), a special-purpose processor, and / or the like.

[0323] I / O subsystem 1808 can include user interface input devices and user interface output devices. User interface input devices can include, for example, a keyboard, a pointing device such as a mouse or trackball, a touch panel or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices can include, for example, a motion sensing and / or gesture recognition device, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with an input device such as the Microsoft Xbox® 360 game controller using gestures and voice commands through a natural user interface. User interface input devices can also include an eye gesture recognition device, such as Google Glass® blink detector, that detects eye activity (e.g., "blinking" of the eye) of a user and converts the eye gestures to input into an input device (e.g., Google Glass®). Further, user interface input devices can include a voice recognition sensing device that enables a user to interact with a voice recognition system (e.g., Siri® navigator) with voice commands.

[0324] User interface input devices can also include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. Additionally, user interface input devices can include, for example, medical imaging input devices such as computerized tomography, magnetic resonance imaging, position emission tomography devices, medical ultrasonic devices. User interface input devices can also include, for example, audio input devices such as MIDI keyboards, digital musical instruments and the like.

[0325] User interface output devices can include a display subsystem, indicator lights, or non- visual displays such as audio output devices, etc. The display subsystem can be a cathode ray tube (CRT), a flat-panel device such as that using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, or any other display device suitable for creating graphical images for user selection. In general, the term "output device" is used in the broadest sense, encompassing all possible types of devices and mechanisms for outputting information from the computer system 1800 to a user or other computer. For example, user interface output devices can include, without limitation, a variety of display devices that visually convey text, graphics, and audio / video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0326] Computer system 1800 may include storage subsystem 1818, which provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1804, provide the aforementioned functionality. Storage subsystem 1818 may also provide a repository for storing data used according to this disclosure.

[0327] like Figure 18 As illustrated in the example, storage subsystem 1818 may include various components, including system memory 1810, computer-readable storage medium 1822, and computer-readable storage medium reader 1820. System memory 1810 may store program instructions that can be loaded and executed by processing unit 1804. System memory 1810 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1810, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0328] System memory 1810 may also store operating system 1816. Examples of operating system 1816 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations of computer system 1800 that execute one or more virtual machines, the virtual machine and its guest operating system (GOS) may be loaded into system memory 1810 and executed by the core of one or more processors or processing units 1804.

[0329] The system memory 1810 can be of various types, depending on the type of computer system 1800. For example, the system memory 1810 can be volatile memory (such as random access memory (RAM)), and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 1810 can include a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1800, such as during start-up.

[0330] The computer readable storage medium 1822 can represent remote, local, fixed, and / or removable storage devices as well as storage media for temporarily and / or more permanently containing, storing, and / or providing computer readable information to and / or for use by the computer system 1800, including instructions that can be executed by the processing unit 1804 of the computer system 1800.

[0331] The computer readable storage medium 1822 can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information such as computer readable instructions, other data structures, program modules, or other data. This can include tangible computer readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media.

[0332] By way of example, computer-readable storage media 1822 can include a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media; a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk; and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD ROM, DVD, and Blu-Ray® disk, or other optical media. Computer-readable storage media 1822 can include, but is not limited to, Zip® drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tape, and the like. Computer-readable storage media 1822 can also include, solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid state ROM, and the like, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetic RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media can provide nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 1800.

[0333] Machine-readable instructions that can be executed by one or more processors or cores of processing unit 1804 can be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include a physical, tangible storage device that comprises volatile memory or non-volatile memory. Examples of non-transitory computer- readable storage media include magnetic storage media (e.g., disk or tape), optical storage media (e.g., DVD or Blu-ray Disc), various types of RAM, ROM, or flash memory, hard disk drives, solid state drives, removable media, or other storage devices.

[0334] The communications subsystem 1824 provides an interface to other computer systems and networks. The communications subsystem 1824 serves as an interface for receiving data from and transmitting data to other systems from the computer system 1800. For example, the communications subsystem 1824 can enable the computer system 1800 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1824 can include a radio frequency (RF) transceiver component (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution) technology, WiFi (IEEE 802.11 family standards), or other mobile communication technologies, or any combination thereof), a Global Positioning System (GPS) receiver component, and / or other components, for use in wired or wireless communication of data. In some embodiments, the communications subsystem 1824 can provide wired communication capabilities (e.g., using Ethernet).

[0335] In some embodiments, the communications subsystem 1824 can also receive input communication in the form of structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, and the like, representing coded information (e.g., internet feeds, Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds,

[0336] As an example, communications subsystem 1824 can be configured to receive data feeds 1826 in real-time from users of social networks and / or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party information sources.

[0337] Further, communications subsystem 1824 can also be configured to receive data in the form of continuous data streams, which can include event streams 1828 and / or event updates 1830 of real-time events that can be continuous in nature or that otherwise do not have explicit terminations, such as sensor data applications, financial ticker applications, network performance measuring applications, clickstream analysis applications, automobile traffic monitoring, and so forth.

[0338] Communications subsystem 1824 can also be configured to output structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, and the like to one or more databases that can be in communication with one or more stream data sources coupled to computer system 1800.

[0339] The computer system 1800 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0340] Due to the ever-evolving nature of computers and networks, the depiction of the computer system 1800 in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0341] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0342] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0343] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It is clearly understood that modifications, both to methods and devices, can be made without departing from the wider spirit and scope of the disclosure as set forth in the following claims. Thus, while specific embodiments have been described in detail, these are not intended to limit the scope of the application, which is to be measured by the claims that follow.

[0344] The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be interpreted in an inclusive sense, com prising both singular and plural, unless otherwise indi cated. The terms "com prises," "com prises," "com prising," and "com prising," as well as the terms "has," "have," "having," and "contains," "containing," are to be construed to be open-ended on their face, unless otherwise specified. The term "connected" is to be construed as partly or wholly contained within, affixed to, or in working contact with the other. Unless otherwise indi cated, the use of the terms "com prises," "com prises," "com prising," and "com prising," in the present disclosure are not used to limit 15 the claims to the process acts described using these terms and / or the structural relations di rectly recited. All methods described herein can be performed in any suitable order unless otherwise indi cated herein or otherwise di rectly contradicted by context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate embodiments of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0345] Disjunctive language such as the phrase "at least one of X, Y, or Z," unless specifically stated otherwise, is otherwise understood within the context as used herein to mean any lit eral com bination of X, Y and / or Z. Therefore, such disjunctive language is generally intended to mean that com prised at least one of X, at least one of Y, or at least one of Z is com prised individually, and / or in any com bination with each other. The terms "com prised of," "com prising," and "com prises" as well as the terms "com prising," "com prises," and "com prised of," are to be construed as open-ended terms (i.e., meaning "including, but not limited to,") unless otherwise indi cated.

[0346] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for practicing the present disclosure. Variations of those preferred embodiments can become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the present disclosure is intended to cover such insubstantial variations as fall within the range of the claims, accordingly. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise contradicted by context.

[0347] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0348] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0349] In the foregoing specification, aspects of the disclosure are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure can be used individually or jointly. Further, embodiments can be utilized in any number of environments and applications beyond the specific embodiments described herein, without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative in nature and not as restrictive. < / set> < / realm>

Claims

1. A method of performing packet-level data-centric protection enforcement in one or more networks, the method comprising: receiving a data packet at an enforcement point (EP) within one or more networks comprising a plurality of enforcement points (EPs); accessing enforcement data, the enforcement data indicating allowed communications between the EP and one or more other EPs, wherein the data is generated from a policy specifying how traffic flows through the one or more networks and a determination of possible data movements between at least two of the plurality of EPs; and enforcing flow of the data packet at the EP based on the data.

2. The method of claim 1, further comprising generating a graph identifying the plurality of EPs and possible data movements between the plurality of EPs, and wherein the enforcement data is generated based at least in part on the graph.

3. The method of any of claims 1-2, further comprising: generating a graph based at least in part on one or more rules in the policy specifying how traffic flows through the enforcement point and other enforcement points, wherein the policy comprises one or more layer 4 rules and one or more layer 7 rules; and wherein the enforcement data is generated based at least in part on the graph.

4. The method of any of claims 1-3, further comprising: assigning each of the plurality of EPs a unique origin ID; and wherein the enforcement data comprises origin IDs of EPs that are neighbors of the EP and an indication of whether to allow communication with each of the neighbors.

5. The method of any of claims 1-4, wherein the enforcement point comprises a network virtualization device (NVD) comprising a smartNIC.

6. The method of any of claims 1-5, wherein enforcing flow of the data packet occurs prior to transmitting the data packet to a next hop.

7. The method of any of claims 1-6, further comprising: determining a source of the data packet based at least in part on a first origin ID; determining a destination of the data packet based at least in part on a second origin ID; and wherein enforcing flow of the data packet comprises preventing the data packet from being transmitted to a next hop based at least in part on one or more of the first origin ID or the second origin ID.

8. The method of any of claims 1-7, further comprising distributing first enforcement data to the EP and second enforcement data to a second EP.

9. A system comprising: one or more networks comprising one or more enforcement points (EPs); a policy specifying how traffic flows through the one or more networks; one or more processors; and a non-transitory computer-readable medium storing a set of instructions which, when executed by the one or more processors, cause performance of processes comprising: receiving a data packet at the EP; ​ ​ accessing enforcement data indicating allowed communications between the EP and one or more other EPs, wherein the data is generated from the policy and a determination of possible data movement between at least two EPs; and enforcing flow of the data packet at the EP based on the data.

10. The system of claim 9, further comprising generating a graph identifying the EPs and possible data movement between the EPs, and wherein the enforcement data is generated based at least in part on the graph.

11. The system of any of claims 9-10, further comprising: generating a graph based at least in part on one or more rules in the policy that specify how traffic flows through the EPs, wherein the policy comprises one or more layer 4 rules and one or more layer 7 rules; and wherein the enforcement data is generated based at least in part on the graph.

12. The system of any of claims 9-11, further comprising: assigning each of the EPs a unique origin ID; and wherein the enforcement data comprises origin IDs of EPs that are neighbors of the EP and an indication of whether communication with each of the neighbors is allowed.

13. The system of any of claims 9-12, wherein the enforcement point comprises a network virtualization device (NVD) that includes a smartNIC.

14. The system of any of claims 9-13, wherein enforcing flow of the data packet occurs prior to transmitting the data packet to a next hop.

15. The system of any of claims 9-14, further comprising: determining a source of the data packet based at least in part on a first origin ID; determining a destination of the data packet based at least in part on a second origin ID; and wherein enforcing flow of the data packet comprises preventing the data packet from being transmitted to a next hop based at least in part on one or more of the first origin ID or the second origin ID.

16. The system of any of claims 9-15, further comprising distributing first enforcement data to the EPs and second enforcement data to a second EP.

17. A computer-readable medium comprising instructions that, when executed, cause one or more processors to perform operations comprising: receiving a data packet at an enforcement point (EP) within one or more networks comprising a plurality of enforcement points (EPs); accessing enforcement data indicating allowed communications between the EP and one or more other EPs, wherein the data is generated from a policy that specifies how traffic flows through the one or more networks and a determination of possible data movement between at least two of the plurality of EPs; and enforcing flow of the data packet at the EP based on the data.

18. A method comprising: receiving a data packet at an enforcement point (EP) within one or more networks comprising a plurality of enforcement points (EPs); accessing enforcement data indicating allowed communications between the EP and one or more other EPs, wherein the data is generated from a policy that specifies how traffic flows through the one or more networks and a determination of possible data movement between at least two of the plurality of EPs; and enforcing flow of the data packet at the EP based on the data.

18. The computer-readable medium of claim 17, further comprising generating a graph that identifies the plurality of EPs and possible data movement between the plurality of EPs, and wherein the enforcement data is generated based at least in part on the graph.

19. The computer-readable medium of any of claims 17-18, further comprising: generating a graph based at least in part on one or more rules in the policy that specify how traffic flows through the enforcement point and other enforcement points, wherein the policy comprises one or more layer 4 rules and one or more layer 7 rules; and wherein the enforcement data is generated based at least in part on the graph.

20. The computer-readable medium of any of claims 17-19, further comprising: assigning each of the plurality of EPs a unique origin ID; and wherein the enforcement data comprises origin IDs of EPs that are neighbors of the EP and an indication of whether communication with each of the neighbors is allowed.

21. The computer-readable medium of any of claims 17-20, wherein the policy comprises one or more rules that specify how traffic is to be handled by the EP.

Images