Automatic generation of role-based access control in cloud on-premise software-defined networking architectures

Abstract

The present disclosure relates to role-based access control auto-generation in cloud-native software-defined networking architectures. A network controller for a software-defined networking (SDN) architecture system can receive a request to generate an access control policy for a role in a container orchestration system, where the request specifies a plurality of functions. The network controller can execute the plurality of functions and can record execution of the plurality of functions in an audit log. The network controller can parse the audit log to determine a plurality of resources of the container orchestration system accessed as a result of executing the plurality of functions and determine, for each resource of the plurality of resources, a respective one or more types of operations performed on the respective resource. The network controller can create the access control policy for the role to permit the role to perform the respective one or more types of operations on each resource of the plurality of resources based at least in part on the parsed audit log.

Description

[0001] Related applications

[0002] This application claims the rights of U.S. non-provisional application No. 17 / 808,970, filed June 24, 2022; U.S. provisional application No. 63 / 362,319, filed May 31, 2022; and Indian provisional application No. 202141044924, filed October 4, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a virtual computing infrastructure, and more specifically, to access control policies for cloud-native networks. Background Technology

[0004] In a typical cloud data center environment, there exists a large collection of interconnected servers providing computing and / or storage capabilities to run various applications. For example, a data center may include facilities hosting applications and services for users (i.e., the data center's customers). For instance, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More sophisticated data centers provide user support facilities located in various physical hosting facilities for infrastructures spread across the globe.

[0005] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers feature widely used virtualized environments in which virtual machines or containers, also known as virtual runtime elements, are deployed on and run on the underlying computing platform of physical computing devices.

[0006] Virtualization within a data center, or any environment comprising one or more servers, offers several advantages. One advantage is the significantly improved efficiency it provides. Virtualization becomes easier and more efficient as the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures where each physical CPU has a large number of cores. A second advantage is the effective control virtualization provides over the computing infrastructure. The provisioning and management of computing infrastructure becomes easier as physical computing resources become replaceable in environments such as cloud-based computing. Therefore, enterprise IT staff often favor virtualized computing clusters in data centers due to their management advantages, in addition to the efficiency and increased ROI offered by virtualization.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable runtime elements for applications that are isolated from each other and from the host machine. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and run as a single lightweight package on any host or virtual machine that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers provide a guarantee of continuous operation from one computing environment to another virtual or physical environment.

[0008] Due to the inherently lightweight nature of containers, a single host can typically support more container instances than a traditional virtual machine (VM). Often, short-lived containers can be created and moved more efficiently than VMs, and containers can also be managed as groups of logically related components (sometimes called "pods" on some orchestration platforms, such as Kubernetes). These container characteristics influence the requirements for container networking solutions: the network should be agile and scalable. VMs, containers, and bare-metal servers may need to coexist in the same compute environment, enabling communication between various deployed applications. Container networks should also be able to work with the diverse types of orchestration platforms used to deploy containerized applications.

[0009] Managing the computing infrastructure used for application deployment and operation likely involves two main roles: (1) orchestration – used to automate the deployment, scaling, and operation of applications across host clusters and to provide the computing infrastructure, which may include container-centric computing infrastructure; and (2) network management – ​​used to create virtual networks within the network infrastructure to enable packetized communication between applications running in virtual runtime environments such as containers or VMs, as well as between applications running in legacy (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0010] In summary, this paper describes a technique for generating role-based access control (RBAC) policies for computer networks implemented using a cloud-native SDN architecture. Users, such as computer network administrators, can request the network controller to generate access control policies for roles by specifying functions provided by the network controller's APIs. The network controller can execute the specified functions and log the execution of these functions in an audit log. The network controller can parse and / or filter the audit logs to extract the resources accessed by executing the functions and the corresponding one or more operations performed on each resource. Therefore, the network controller can generate access control policies for roles based on the parsed and / or filtered audit logs, allowing roles to perform the corresponding one or more operations on each resource.

[0011] In some examples, an SDN architecture for a computer network may include data plane elements implemented in compute nodes, as well as network devices such as routers or switches. The SDN architecture may also include a network controller for creating and managing virtual networks. The SDN architecture configuration and control plane is designed to support scaling cloud-native software with a container-based microservices architecture that supports upgrades within services. Configuration nodes can be implemented to expose custom resources. These custom resources configured in the SDN architecture may include configuration elements typically exposed by the network controller; however, configuration elements can be merged with Kubernetes-native / built-in resources to support a unified intent model exposed by an aggregated API layer implemented by the Kubernetes controller and a custom resource controller dedicated to reconciling the actual state of the SDN architecture to the intended state.

[0012] Computer networks implemented using SDN architectures can be complex environments comprising hundreds or thousands of physical and / or virtual components that communicate with each other and with external devices, such as applications, virtual machines, virtual routers, virtual networks, subnets, domains, tenants, resources, etc. Furthermore, resources can be hierarchical in nature, so accessing a resource may also require accessing dependent resources. Therefore, it may be impractical for users (e.g., administrators) to manually determine appropriate access control policies for roles to execute one or more workflows.

[0013] In some examples, if a user attempts to manually configure a role's access policy, the user may over- or under-enclose in determining the operations a role can perform on resources. If the user over-encloses a role's access policy, they may grant the role the ability to perform operations on resources that the role might not need to access API functionality required for its work, potentially leading to security issues. In other examples, the user may mistakenly deny a role the ability to perform one or more necessary operations to access API functionality that the role might need, thereby potentially preventing the user from successfully enabling the role to perform its duties.

[0014] Therefore, the techniques described herein can provide one or more technical advantages that lead to at least one practical application. For example, by creating access policies based on API functionality, a network controller of a computer network can create access policies for roles to prevent roles from performing operations on resources within the computer network that do not require the specified functions. Because the techniques described herein can improve the security of a computer network by granting roles permission to perform operations on resources to execute one or more functions of an API, they prevent the erroneous or accidental granting of permissions to roles to perform operations on resources that may not be necessary for the specified functions.

[0015] In some aspects, the technology described herein relates to a network controller for a software-defined networking (SDN) architecture system, the network controller comprising: a processing circuit; and one or more configuration nodes configured to be executed by the processing circuit, wherein the one or more configuration nodes include an application programming interface (API) server to handle requests for operations on local resources of a container orchestration system, and the one or more configuration nodes include a custom API server to handle requests for operations on custom resources configured with the SDN architecture; to: receive requests to generate access control policies for roles in the container orchestration system, wherein the requests specify multiple functions provided by the custom API server and an aggregate API provided by the API server; execute the multiple functions; log the execution of the multiple functions in an audit log; parse the audit log to determine multiple resources of the container orchestration system accessed due to the execution of the multiple functions and, for each of the multiple resources, determine corresponding one or more types of operations for multiple actions performed on the corresponding resource due to the execution of the multiple functions; and create access control policies for roles based at least in part on the parsed audit logs to permit roles to perform corresponding one or more types of operations on each of the multiple resources.

[0016] In some aspects, the technology described herein relates to a method comprising: receiving a request by processing circuitry of a network controller for a software-defined networking (SDN) architecture system to generate an access control policy for a role in a container orchestration system, wherein the network controller includes an application programming interface (API) server to handle requests for operations on local resources of the container orchestration system, and the network controller includes a custom API server to handle requests for operations on custom resources configured with the SDN architecture, and wherein the request specifies multiple functions of an aggregated API provided by the custom API server and the API server; the processing circuitry executing the multiple functions; the processing circuitry logging the execution of the multiple functions in an audit log; the processing circuitry parsing the audit log to determine multiple resources of the container orchestration system accessed due to the execution of the multiple functions and, for each of the multiple resources, determining a corresponding one or more types of operation for a plurality of actions performed on the respective resource due to the execution of the multiple functions; and the processing circuitry creating an access control policy for a role, at least in part based on the parsed audit log, to permit the role to perform a corresponding one or more types of operation on each of the multiple resources.

[0017] In some aspects, the technology described herein relates to a non-transitory computer-readable medium comprising instructions for causing processing circuitry of a network controller to execute one or more configuration nodes, the one or more configuration nodes including an application programming interface (API) server to handle requests for operations on local resources of a container orchestration system, and the one or more configuration nodes including a custom API server to handle requests for operations on custom resources of an SDN architecture configuration, to: receive a request to generate an access control policy for a role in the container orchestration system, wherein the request specifies multiple functions of an aggregate API provided by the custom API server and the API server; execute the multiple functions; log the execution of the multiple functions in an audit log; parse the audit log to determine multiple resources of the container orchestration system accessed due to the execution of the multiple functions and, for each of the multiple resources, determine a corresponding one or more types of operation for a plurality of actions performed on the corresponding resource due to the execution of the multiple functions; and create an access control policy for a role, at least in part based on the parsed audit log, to permit the role to perform a corresponding one or more types of operation on each of the multiple resources.

[0018] Details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objectives, and advantages will become apparent from the description and drawings, and from the claims. Attached Figure Description

[0019] Figure 1 This is a block diagram illustrating an exemplary computing infrastructure in which examples of the technologies described herein can be implemented.

[0020] Figure 2 This is a block diagram illustrating an example of a cloud-native SDN architecture for cloud-native networks according to the technology disclosed herein.

[0021] Figure 3 This is a block diagram that further illustrates in detail another view of the components of a cloud-native SDN architecture according to the technology disclosed herein.

[0022] Figure 4 This is a block diagram illustrating exemplary components of a cloud-native SDN architecture according to the technology disclosed herein.

[0023] Figure 5 This is a block diagram of an exemplary computing device based on the technology described in this disclosure.

[0024] Figure 6 This is a block diagram of an exemplary computing device for computing nodes of one or more clusters in a cloud-native SDN architecture system, operating according to the technology disclosed herein.

[0025] Figure 7A This is a block diagram illustrating the control / routing plane for configuring the underlying network and overlay network using a cloud-native SDN architecture, according to the technology disclosed herein.

[0026] Figure 7B This is a block diagram illustrating a configured virtual network that uses a tunnel connection to a container pool in the underlying network according to the technology disclosed herein.

[0027] Figure 8 This is a block diagram illustrating an example of a custom controller for configuring custom resources for a cloud-native SDN architecture according to the techniques disclosed herein.

[0028] Figure 9 This is a block diagram illustrating an exemplary flow for creating, monitoring, and coordinating custom resource types that depend on different custom resource types.

[0029] Figure 10 This is a block diagram illustrating the virtualization of access control policies, roles, and exemplary relationships between subjects.

[0030] Figure 11 This is a flowchart illustrating exemplary operation of a network controller in an SDN architecture according to the technology disclosed herein.

[0031] Similar reference numerals are used to identify similar elements throughout the description and figures. Detailed Implementation

[0032] Figure 1 This is a block diagram of an exemplary computing infrastructure 8 illustrating examples of technologies that can be implemented as described herein. Current implementations of software-defined networking (SDN) architectures for virtual networks face challenges in cloud-native adoption due to factors such as the complexity of lifecycle management, the high mandatory nature of resource analysis components, the limited scalability of configuration modules, and the lack of command-line interface (CLI)-based interfaces (such as kubectl). Computing infrastructure 8 includes the cloud-native SDN architecture system described herein, which addresses these challenges and modernizes the era of telecom cloud-native architecture. Exemplary use cases for cloud-native SDN architecture include 5G mobile networks as well as cloud and enterprise cloud-native use cases. The SDN architecture may include data plane elements implemented in compute nodes (e.g., server 12) and network devices such as routers or switches, and may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. The SDN architecture configuration and control plane is designed with horizontally scaled cloud-native software featuring a container-based microservices architecture that supports hot upgrades.

[0033] Therefore, SDN architecture components are microservices, and unlike existing network controllers, SDN architecture assumes a foundational container orchestration platform for managing the lifecycle of SDN architecture components. The container orchestration platform is used to build SDN architecture components; SDN architecture uses cloud-native monitoring tools that can integrate with customer-provided cloud-native options; SDN architecture uses an aggregated API for SDN architecture objects to provide a declarative approach to resources (i.e., custom resources). SDN architecture upgrades can follow a cloud-native model, and SDN architecture can leverage Kubernetes constructs such as Multus, authentication & authorization, cluster API, KubeFederation, KubeVirt, and Kata containers. SDN architecture can support Data Plane Development Kit (DPDK) container pools, and SDN architecture can extend to support Kubernetes with virtual network policies and global security policies.

[0034] For service providers and enterprises, SDN architecture automates network resource configuration and orchestration to dynamically create highly scalable virtual networks and link Virtualized Network Functions (VNFs) with Physical Physical Functions (PNFs) to form differentiated service chains on demand. SDN architecture can be integrated with orchestration platforms (e.g., orchestrators23) such as Kubernetes, OpenShift, Mesos, OpenStack, and VMware vSphere, and can also be integrated with service provider operations support systems / business support systems (OSS / BSS).

[0035] Typically, one or more data centers 10 provide an operating environment for applications and services at a customer site 11 (shown as "Customer 11"), which has one or more customer networks coupled to the data centers via a service provider network 7. For example, each data center 10 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 15, which may represent one or more networks managed by other vendors, and thus may form part of a large-scale public network infrastructure (e.g., the Internet). For example, the public network 15 may represent a local area network (LAN), wide area network (WAN), Internet, virtual LAN (VLAN), corporate LAN, Layer 3 virtual private network (VPN), or Internet Protocol (IP) intranet operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0036] Although customer site 11 and public network 15 are primarily shown and described as the edge network of service provider network 7, in some embodiments, customer site 11 and one or more networks in public network 15 can be tenant networks within any data center 10. For example, data center 10 may host multiple tenants (customers), each associated with one or more virtual private networks (VPNs), where each tenant may implement one or more customer sites 11.

[0037] Service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 15. Service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. Service provider network 7 may implement Multiprotocol Label Switching (MPLS) forwarding and, in this instance, may be referred to as an MPLS network or MPLS backbone. In some instances, service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, providing services from one or more service providers.

[0038] In some embodiments, each data center 10 may represent one of a plurality of geographically distributed network data centers, which may be connected to each other via a service provider network 7, a dedicated network link, a black fiber optic cable, or other connections. Figure 1 As illustrated in the example, data center 10 may include facilities that provide network services to customers. Customers of the service provider may be entities such as businesses and governments or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although shown as a discrete edge network of service provider network 7, elements in data center 10, such as one or more physical network functions (PNFs) or virtualized network functions (VNFs), may be included within the core of service provider network 7.

[0039] In this example, data center 10 includes storage and / or compute servers (or “nodes”) interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, and servers 12A-12X (here referred to as “server 12”) are described as coupled to top-of-rack TOR switches 16A-16N. Server 12 is a compute device and may also be referred to herein as a “compute node,” “host,” or “host device.” Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail; however, data center 10 may include multiple additional servers coupled to other TOR switches 16 of data center 10.

[0040] In the example shown, switching architecture 14 includes interconnecting top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") to the distribution layer of chassis (or "spine" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, however, data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Data center 10 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0041] In this example, TOR switch 16 and chassis switch 18 provide redundant (multihomed) connectivity to server 12 to IP fabric 20 and service provider network 7. Chassis switch 18 aggregates traffic flows and provides connectivity between TOR switches 16. TOR switch 16 may be a network device providing Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching capabilities. TOR switch 16 and chassis switch 18 may each include one or more processors and memory and are capable of running one or more software processes. Chassis switch 18 is coupled to IP fabric 20, which can perform Layer 3 routing to route network traffic between data center 10 and customer site 11 via service provider network 7. The switching architecture of data center 10 is merely an example. For example, other switching architectures may have more or fewer switching layers. IP fabric 20 may include one or more gateway routers.

[0042] The term "packet flow," "traffic flow," or simply "flow" refers to a collection of packets that originate from a specific source device or endpoint and are sent to a specific destination device or endpoint. For example, a single flow of packets can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, this 5-tuple identifies the packet flow corresponding to a received packet. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to either <source network address, destination network address> or <source network address, source port>.

[0043] Each of the servers 12 can represent a compute server or a storage server. For example, each server 12 can represent a computing device configured to operate according to the technologies described herein, such as an x86 processor-based server. The servers 12 can provide Network Functions Virtualization Infrastructure (NFVI) for an NFV architecture.

[0044] Virtualization of server resources allows any of the 12 servers to be configured with virtual execution elements such as container pools or virtual machines, providing a degree of isolation between one or more processes (applications) running on the server. "Hypervisor-based," "hardware-level," or "platform" virtualization refers to the creation of virtual machines, each including a guest operating system for running one or more processes. Typically, virtual machines provide a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are visualized from the host server's physical hardware, the executing applications are isolated from the host hardware and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communication on a corresponding virtual network.

[0045] A virtual network is a logical construct implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in virtualized data centers, such as data center 10. Each tenant or application can have one or more virtual networks. Unless explicitly permitted by security policies, each virtual network must be isolated from all other virtual networks.

[0046] Virtual networks can use data center 10 gateway routers ( Figure 1 (Not shown) Connects to and extends to Physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs). Virtual networks can also be used to implement Network Functions Virtualization (NFV) and service links.

[0047] Virtual networks can be implemented using various mechanisms. For example, each virtual network can be implemented as a Virtual Local Area Network (VLAN), a Virtual Private Network (VPN), etc. Virtual networks can also be implemented using two networks—a physical underlying network consisting of IP structure 20 and switching structure 14, and a virtual overlay network. The role of the physical underlying network is to provide the "IP structure," which provides unicast IP connections from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0048] As further described below with respect to virtual router 21 (shown and referred to herein as "vRouter21"), the virtual router running in server 12 creates a virtual overlay network over the physical underlying network using its own dynamic "tunneling" mesh. For example, these overlay tunnels can be MPLS over GRE / UDP tunnels, VXLAN tunnels, or NVGRE tunnels. For virtual machines or other virtual execution elements, the underlying physical routers and switches cannot store arbitrary state for each tenant, such as arbitrary Media Access Control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches can only contain the IP prefix or MAC address of physical server 12. (The gateway router or switch connecting the virtual network to the physical network is an exception and can contain tenant MAC or IP addresses.)

[0049] The virtual router 21 of server 12 typically contains the state for each tenant. For example, a virtual router may contain a separate forwarding table (routing instance) for each virtual network. This forwarding table contains the IP prefixes (in the case of Layer 3 coverage) or MAC addresses (in the case of Layer 2 coverage) of the virtual machines or other virtual execution elements (e.g., a container pool of containers). A single virtual router 21 does not need to contain all the IP prefixes or MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain these routing instances that exist locally on server 12 (i.e., those with at least one virtual execution element present on server 12).

[0050] "Container-based" or "OS-based" virtualization refers to the virtualization of an operating system running multiple isolated systems on a single machine (virtual or physical). These isolated systems represent containers such as those provided by the open-source DOCKER container application or CoreOS Rkt ("Rocket"). Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can forgo its own operating system and instead provide an application suite and application-specific libraries. Typically, containers are run by the host as isolated user-space instances, and they can share the operating system and common libraries with other containers running on the host. Therefore, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communication on a corresponding virtual network.

[0051] In some examples, containers are managed by their host kernel to allow for the limitation and prioritization of resources (CPU, memory, block I / O, network, etc.) using namespace isolation in certain situations, without launching arbitrary virtual machines. Namespace isolation allows for complete isolation of the application's (e.g., given a container's) view of the operating environment, including process trees, networking, user identifiers, and mounted file systems. In some embodiments, containers can be deployed using Linux Containers (LXC), an operating system-level virtualization approach that uses a single Linux kernel to run multiple isolated Linux systems (containers) on a controlling host.

[0052] Server 12 hosts virtual network endpoints of one or more virtual networks operating on the physical network represented here by IP structure 20 and switching structure 14. Although described primarily in contrast to a data center-based switching network, other physical networks such as service provider network 7 may reside beneath the one or more virtual networks.

[0053] Each server 12 can host one or more virtual execution elements, each having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements sharing a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, one or more sets of containers (e.g., a container pool), or another virtual execution element such as a Layer 3 endpoint of a virtual network. The term "virtual execution element" encompasses virtual machines, containers, and other virtualized computing resources that provide at least partially independent runtime environments for applications. The term "virtual execution element" can also encompass a container pool of one or more containers. A virtual execution element can represent an application workload. Figure 1 As shown, server 12A hosts a virtual network endpoint as a container pool 22 with one or more containers. However, in practice, given the hardware resource limitations of server 12, server 12 can function as multiple virtual execution elements. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or other processing on packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0054] Each of the servers 12 includes at least one network interface card (NIC) 13, each NIC 13 including at least one interface for exchanging packets with the TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC (“physical function”). For example, in Single Root I / O Virtualization (SR-IOV) as described in the Peripheral Component Interface Special Interest Group (SR-IOV) specification, the PCIe physical function of a network interface card (or “network adapter”) is virtualized to provide one or more virtual network interfaces as “virtual functions” for use by corresponding endpoints running on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources and virtual functions are examples of virtual hardware components 21. As another example, one or more servers 12 can implement, for example, Virtio, a paravirtualization framework available on the Linux operating system, which provides emulated NIC functionality as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement Open vSwitch to perform distributed virtual multi-tier switching between one or more virtual NICs (vNICs) of the hosted virtual machines, where the vNIC may also represent a type of virtual hardware component that provides a virtual network interface to virtual network endpoints. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function. In some examples, any of the servers 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interface. For a Docker implementation of containers hosted by the server, a Linux bridge or other operating system bridge running on the server and switching packets between containers may be referred to as a "Docker bridge". As used herein, the term "virtual router" may encompass a Contrail or Tungsten-structured virtual router, an Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software located on a host device that perform switching, bridging, or routing of packets between virtual network endpoints in one or more virtual networks, hosted by one or more servers 12 in the virtual network.

[0055] Any NIC 13 may include an internal device switch that switches data between virtual hardware components associated with the NIC. For example, for an SR-IOV-enabled NIC, the internal device switch may be a virtual Ethernet bridge (VEB) that switches between SR-IOV virtual functions and correspondingly between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. Alternatively, the internal device switch may be referred to as a NIC switch, or, in the case of an SR-IOV implementation, an SR-IOV NIC switch. The virtual hardware components associated with NIC 13A may be associated with a Layer 2 destination address assigned by NIC 13A or the software process responsible for configuring NIC 13A. Physical hardware components (or “physical functions” in the SR-IOV implementation) are also associated with a Layer 2 destination address.

[0056] One or more servers 12 may each include a virtual router 21 that performs one or more routing instances for a corresponding virtual network within data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a Virtual Router and Forward Instance (VRF) of an Internet Protocol Virtual Private Network (IP-VPN). For example, packets received by the virtual router 21 of server 12A from the underlying physical network infrastructure (i.e., IP infrastructure 20 and switching infrastructure 14) of data center 10 may include an external header to allow the physical network infrastructure to tunnel the payload or “internal packet” to the physical network address of the network interface card 13A of server 12A performing the virtual router. The external header may include not only the physical network address of the server’s network interface card 13A, but also a virtual network identifier such as a VxLAN tag or Multiprotocol Label Switching (MPLS) tag identifying a virtual network and the corresponding routing instance performed by the virtual router 21. The internal packet includes an internal header with a destination network address that corresponds to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0057] Virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the packet's tunnel encapsulation header, and forwards the packet to the appropriate destination virtual network endpoint. For example, for server 12A, for each packet outbound from a virtual network endpoint (e.g., container pool 22) hosted by server 12A, virtual router 21 appends a tunnel encapsulation header indicating the virtual network to the packet to generate an encapsulated or "tunnel" packet, and virtual router 21 outputs the encapsulated packet via the virtual network overlay tunnel to a physical destination computing device such as another server 12. As used herein, virtual router 21 can perform tunnel endpoint operations to encapsulate internal packets originating from a virtual network endpoint to generate a tunnel packet, and decapsulate the tunnel packet to obtain internal packets for routing to other virtual network endpoints.

[0058] In some examples, the virtual router 21 may be kernel-based and may run as part of the kernel of the operating system of server 12A.

[0059] In some examples, virtual router 21 may be a virtual router that supports the Data Plane Development Kit (DPDK). In this example, virtual router 21 uses DPDK as its data plane. In this mode, virtual router 21 runs as a user-space application linked to a DPDK library (not shown). This is an implementation of a virtual router and is typically used by telecommunications companies, where the VNF is usually a DPDK-based application. Virtual router 21, as an implementation of a DPDK virtual router, can achieve up to ten times the throughput of a virtual router operating as a kernel-based virtual router. It uses a physical interface powered by DPDK's polling-mode driver (PMD) instead of the interrupt-based driver of the Linux kernel.

[0060] User-I / O (UIO) kernel modules such as vfio or uio_pci_generic can be used to expose the registers of the physical network interface to user space, making them accessible to the DPDK PMD. When NIC 13A is bound to the UIO driver, NIC 13A is moved from the Linux kernel space to user space and is therefore no longer managed or visible by the Linux OS. Thus, the DPDK application (i.e., virtual router 21A in this example) fully manages NIC 13A. This includes packet polling, packet processing, and packet forwarding. User packet processing steps can be performed by the DPDK data plane of virtual router 21A, while the kernel (in...) Figure 1(The kernel is not shown in the diagram) Participation is limited or non-participation. Compared to interrupt mode, specifically when packet rates are high, the nature of this "polling mode" makes packet processing / forwarding in the virtual router 21DPDK data plane more efficient. Limited interrupts or no interrupts are present during packet I / O, and context switching occurs.

[0061] Additional details about the DPDK vRouter example can be found in “DAY ONE: CONTRAIL DPDKvROUTER” by Kiran KN et al. of Juniper Networks in 2021, which is combined here in its entirety by reference.

[0062] The compute infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12, providing a virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration system that provides container-centric infrastructure by automating the deployment, scaling, and operation of containers, thereby providing container-centric infrastructure. In the context of virtualized compute infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services running on host servers available to the orchestration platform. Specifically, for example, container orchestration licenses container coordination and refers to the deployment, management, scaling, and configuration of containers to host servers by the container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes (a container orchestration system), Docker Swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0063] The components of the automation platform for computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework, in which the cluster master node manages the deployment and operation of containers to one or more cluster slave nodes. The terms "master node" and "slave node" used here encompass different orchestration platform terms that distinguish the primary management element of a cluster from the analog device that hosts the primary container in the cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "slave node," while the Docker Swarm platform refers to both the cluster manager and cluster nodes.

[0064] The orchestrator 23 and network controller 24 can run on separate computing devices or on the same computing device. Each of the orchestrator 23 and network controller 24 can be a distributed application running on one or more computing devices. The orchestrator 23 and network controller 24 can implement a corresponding master node for one or more clusters, each of which has one or more slave nodes (also referred to as "compute nodes") implemented by a corresponding server 12.

[0065] Typically, for example, network controller 24 controls the network configuration of the structure of data center 10 to establish one or more virtual networks for packetized communication between virtual network endpoints. Network controller 24 provides a logically centralized and, in some cases, physically centralized controller for operating one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or administrator / operator. Additional information regarding exemplary operation of network controller 24 in conjunction with other devices or other software-defined networks of data center 10 is found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled “PHYSICAL PATH DETERMINATION FOR VIRTUALNETWORK PACKET FLOWS”, and in U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled “Tunneled Packet Aggregation for Virtual Networks,” which are fully described herein and incorporated herein by reference.

[0066] Typically, orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides the computing infrastructure, which may include container-centric computing infrastructure. Orchestrator 23, and in some cases, network controller 24, can implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud can provide virtualization infrastructure to the container management platform. See below for reference. Figure 3 Describe exemplary components of the Kubernetes orchestration system.

[0067] In one example, container pool 22 is a Kubernetes container pool and an example of a virtual network endpoint. A container pool is a group of one or more logically related containers. Figure 1(Not shown in the diagram), shared memory for containers, and options regarding how to run containers. Alternatively, if instantiated for execution, a container pool may be referred to as a "container pool replica." Each container in container pool 22 is an example of a virtual execution element. Containers in a container pool always reside together on a single server, are co-scheduled, and run in a shared context. The shared context of a container pool can be a collection of Linux namespaces, cgroups, and other isolation aspects. Within the context of a container pool, individual applications may further apply sub-isolation. Typically, containers within a container pool share a common IP address and port space and can detect each other via the localhost. Because they have a shared context, containers within a container pool also communicate with each other using inter-process communication (IPC). Examples of IPC include System V semaphores or POSIX shared memory. Typically, containers that are members of different container pools have different IP addresses, and containers cannot communicate via IPC without the configuration to enable this feature. Instead, containers that are members of different container pools typically communicate with each other via the container pool IP address.

[0068] Server 12A includes a container platform 19 for running containerized applications such as those in container pool 22. Container platform 19 receives requests from orchestrator 23 to obtain containers and host them in server 12A. Container platform 19 then obtains and runs the containers.

[0069] The Container Network Interface (CNI) 17 configures virtual network interfaces for virtual network endpoints. The orchestrator 23 and container platform 19 use CNI 17 to manage the network of container pools (including container pool 22). For example, CNI 17 creates virtual network interfaces that connect container pools to virtual router 21 and enable containers in that container pool to communicate with other virtual network endpoints on the virtual network via these virtual network interfaces. For example, CNI 17 can insert virtual network interfaces for virtual networks into the network namespaces of containers in container pool 22 and configure (or request configuration) virtual network interfaces for virtual networks in virtual router 21 such that virtual router 21 is configured to send packets received from the virtual network via the virtual network interface to containers in container pool 22 and to send packets received via the virtual network interface on the virtual network from containers in container pool 22. CNI 17 can assign network addresses (e.g., virtual IP addresses for virtual networks) and can configure routes for virtual network interfaces. In Kubernetes, by default, all container pools can communicate with all other container pools without using Network Address Translation (NAT). In some cases, orchestrator 23 and network controller 24 create service virtual networks and container pool virtual networks shared by all namespaces, allocating service and container pool network addresses from the namespaces respectively. In other cases, all container pools generated in the Kubernetes cluster across all namespaces can communicate with each other, and network addresses for all container pools can be allocated from the container pool subnet specified by orchestrator 23. When a user creates an isolated namespace for a container pool, orchestrator 23 and network controller 24 can create new container pool virtual networks and new shared service virtual networks for the new isolated namespace. Container pools generated in the Kubernetes cluster within the isolated namespace obtain network addresses from the new container pool virtual network, and the corresponding services of that container pool obtain network addresses from the new service virtual network.

[0070] CNI 17 can represent a library, plugin, module, runtime, or other executable code of server 12A. CNI 17 can be at least partially compliant with the Container Network Interface (CNI) specification or the rkt network proposal. CNI 17 can represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNIs. Alternatively, CNI 17 can be referred to as a network plugin, CNI plugin, or CNI instance. For example, a separate CNI can be called by a Multus CNI to establish different virtual network interfaces for container pool 22.

[0071] CNI 17 can be invoked by orchestrator 23. For the purposes of the CNI specification, containers can be considered to be synchronized with the Linux network namespace. The corresponding unit depends on the specific container runtime implementation: for example, in implementations of application container specifications such as rkt, each container pool runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a uniquely addressable set of entities that can communicate with each other. This can be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can be conceptually added to or removed from one or more networks. The CNI specification defines many considerations for consistent plugins (“CNI plugins”).

[0072] Container pool 22 includes one or more containers. In some examples, container pool 22 includes a containerized DPDK workload designed to use DPDK to speed up packet processing, for example, by exchanging data with other components using the DPDK library. In some examples, virtual router 21 may run as a containerized DPDK workload.

[0073] Container pool 22 is configured with a virtual network interface 26 for sending and receiving packets via virtual router 21. Virtual network interface 26 can be the default interface of container pool 22. Container pool 22 can implement virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while virtual router 21 can implement virtual network interface 26 as a tap interface, virtio user interface, or other types of interface.

[0074] Container pool 22 and virtual router 21 exchange data packets using virtual network interface 26. Virtual network interface 26 can be a DPDK interface. Container pool 22 and virtual router 21 can configure virtual network interface 26 using vhost. Container pool 22 can operate according to an aggregation model. Container pool 22 can use virtual devices, such as virtio devices with vhost-user adapters, for inter-process communication between user-space containers on virtual network interface 26.

[0075] CNI 17 can be combined Figure 1 One or more other components shown configure virtual network interface 26 for container pool 22. Any container in container pool 22 can utilize (i.e., share) the virtual network interface 26 of container pool 22.

[0076] Virtual network interface 26 may represent a virtual Ethernet (“veth”) pair, wherein each end of the pair is an independent device (e.g., a Linux / Unix device), and one end of the pair is assigned to container pool 22 and the other end is assigned to virtual router 21. A veth pair, or one end of a veth pair, is sometimes referred to as a “port”. A virtual network interface may represent a macvlan network with Media Access Control (MAC) addresses assigned to container pool 22 and virtual router 21 for communication between containers in container pool 22 and virtual router 21. Alternatively, for example, a virtual network interface may be referred to as a Virtual Machine Interface (VMI), container pool interface, container network interface, tap interface, veth interface, or simply a network interface (in the specific context).

[0077] exist Figure 1 In the exemplary server 12A, container pool 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment that specifies virtual networks and designates container pool 22 (or one or more containers therein) as a virtual network endpoint of the virtual networks. For example, orchestrator 23 may receive configuration data from users, operators / administrators, or other machine systems.

[0078] As part of the process of creating container pool 22, orchestrator 23 requests network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). Container pool 22 may have different virtual network interfaces for each virtual network it belongs to. For example, virtual network interface 26 may be a virtual network interface for a specific virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks. Network controller 24 processes the request to generate interface configuration data for the virtual network interfaces of container pool 22. Interface configuration data may include a container or container pool unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. Network configuration data for virtual network interfaces may include network name, assigned virtual network address, MAC address, and / or domain name server value. Below is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0079] Network controller 24 sends interface configuration data to server 12A, and more specifically, in some cases, to virtual router 21. To configure virtual network interfaces for container pool 22, orchestrator 23 can invoke CNI 17. CNI 17 obtains and processes the interface configuration data from virtual router 21. CNI 17 creates the individual virtual network interfaces specified in the interface configuration data. For example, CNI 17 can attach one end of a veth pair implementing management interface 26 to virtual router 21 and can attach the other end of the same veth pair implementing management interface 26 using a virtio user to container pool 22.

[0080] Below is an example interface configuration data for the virtual network interface 26 used in container pool 22.

[0081]

[0082]

[0083] A typical CNI plugin is invoked by the container platform / runtime, receives an Add command from the container platform to add a container to a single virtual network, and can subsequently be invoked to receive a Del command from the container / runtime and remove the container from the virtual network. The term "invocation" can refer to the instantiation of a software component or module in memory as executable code to be executed by processing circuitry.

[0084] Network controller 24 is a cloud-native distributed network controller for software-defined networking (SDN) implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 can be implemented using one or more cloud-native component microservices. Each control node 32 can be implemented using one or more cloud-native component microservices.

[0085] In some examples, and as described in further detail below, configuration node 30 can be implemented by extending the local orchestration platform to support custom resources for the orchestration platform used for software-defined networking. More specifically, for example, a northbound interface can be provided to the orchestration platform to support the purpose-driven / declarative creation and management of virtual networks by configuring virtual network interfaces of virtual execution elements, configuring the underlying network connectivity server 12, and configuring overlay routing functions including overlay tunnels for virtual networks and overlay trees for multicast Layer 2 and Layer 3.

[0086] As Figure 1As part of the SDN architecture shown, network controller 24 can be multi-tenant-aware and support multi-tenancy across orchestration platforms. For example, network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. Network controller 24 can also support Kubernetes-defined network constructs and advanced network features such as virtual networks, BGPaaS, network policies, service chains, and other telecommunications features. Network controller 24 can support network isolation using virtual network constructs and can support Layer 3 networking.

[0087] To interconnect multiple virtual networks, network controller 24 can use (and configure in the underlying and / or virtual router 21) a network policy known as a Virtual Network Policy (VNP), and alternatively referred to herein as a Virtual Network Router or Virtual Network Topology. A VNP defines the connectivity policies between virtual networks. A single network controller 24 can support multiple Kubernetes clusters, and thus, VNPs allow for the connection of multiple virtual networks within namespaces, Kubernetes clusters, and across Kubernetes clusters. VNPs can also be extended to support virtual network connectivity across multiple instances of network controller 24.

[0088] Network controller 24 can support multi-layered security using network policies. Kubernetes' default behavior is for container pools to communicate with each other. To apply network security policies, the SDN architecture implemented by network controller 24 and virtual router 21 can operate as a Kubernetes CNI via CNI 17. For Layer 3, network-level isolation occurs and virtual networks operate at L3. Virtual networks are connected via policies. Kubernetes native network policies provide Layer 4 security. The SDN architecture can support Kubernetes network policies. Kubernetes network policies operate at the boundaries of Kubernetes namespaces. The SDN architecture can add custom resources to enhance network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta tags to apply granular security policies in an extensible manner). In some examples, for Layer 4+, the SDN architecture can support integration with containerized security appliances and / or Istio and can provide cryptographic support.

[0089] As Figure 1As part of the SDN architecture shown, network controller 24 can support multi-cluster deployments that are critical for telecom cloud and high-end enterprise use cases. For example, the SDN architecture can support multiple Kubernetes clusters. The cluster API can be used to support lifecycle management of Kubernetes clusters. KubefedV2 can be used to configure node 30 federation across Kubernetes clusters. The cluster API and KubefedV2 are optional components for supporting a single instance of network controller 24, which supports multiple Kubernetes clusters.

[0090] SDN architectures can provide insights into infrastructure, clustering, and applications using network user interfaces and telemetry components. Telemetry nodes can be cloud-native and include microservices that support these insights.

[0091] Role-based access control (RBAC) can be a role-based technology for restricting and monitoring user access within computer infrastructure 8. Computer infrastructure 8 (e.g., via network controller 24) can be configured with one or more roles, each of which can be assigned to one or more users and / or services. The roles assigned to users or services can determine the services offered to the user or service, the applications the user or service is permitted to access within computer infrastructure 8, the administrative privileges the user or service enjoys within computer infrastructure 8, or any combination thereof. Each role can be associated with an access control policy that specifies the associated role's permissions to perform specific operations and / or access specific objects and / or resources within computer infrastructure 8. For example, a role's access control policy can specify one or more objects and / or resources within computer infrastructure 8 that allow the role to perform one or more operations, and can also specify one or more operations that the role is permitted to perform on each of the one or more objects and / or resources specified in the policy.

[0092] In some examples, a role's access control policy can specify one or more of the create, read, update, and delete (CRUD) operations that a role is permitted to perform on one or more objects or resources in computer infrastructure 8. In some instances, the access control policy can act as a whitelist, specifying objects and / or operations that a role can access and / or perform, but preventing the specified objects and / or operations that a role cannot access. For example, if the access control policy does not specify a specific object within computer infrastructure 8, the role associated with the access control policy cannot perform arbitrary operations on the object. In another example, if the access control policy specifies one or more operations that a role can perform on objects within computer infrastructure 8, but does not specify the specific operations that the role can perform on the object, then the role associated with the access control policy can only perform one or more specified operations on the object, but cannot perform the specific operations on the object.

[0093] In some examples, administrators (e.g., users) can manually create and / or modify access control policies for roles to specify the objects that roles are permitted to access within computer infrastructure 8 and the operations that permitted roles can perform on those specified objects. Administrators can interact with a user interface, such as a graphical user interface presented by providing user input at a user interface device to specify permissions for roles to access objects and perform operations within computer infrastructure 8. For example, an administrator can provide user input to select one or more objects within computer infrastructure 8 for which a permitted role can perform one or more operations, and can provide input specifying one or more operations, such as one or more CRUD operations, for each object selected by the administrator.

[0094] Because administrators can easily specify roles that allow cloud administrators to perform all CRUD operations on all objects in, for example, computer infrastructure 8, it is relatively easy and straightforward to manually create access control policies for general high-level roles (such as cloud administrators or tenant administrators). However, in the case of cloud administrators or specific tenant roles (in the case of tenant administrators), it is difficult for administrators to manually create more granular access control policies at the individual object level in computer infrastructure 8.

[0095] For example, some functions performed by computer infrastructure 8 may include performing operations on dozens, hundreds, or thousands of different objects and resources within computer infrastructure 8. Therefore, the administrator needs to manually select access control policies for dozens, hundreds, or thousands of objects and resources within computer infrastructure 8 to create access control policies for roles performing that function. Furthermore, because users in computer infrastructure 8 can specify the functions performed by computer infrastructure 8 as one or more user intents—that is, high-level descriptions of end functions of computer infrastructure 8—the users of computer infrastructure 8 may not be aware of the full range of operations performed by computer infrastructure 8 to execute those one or more user intents.

[0096] Furthermore, specific resources and objects can be hierarchical; therefore, a role can only be permitted to perform CRUD operations on a resource if it may perform the same CRUD operations on dependent resources related to that resource hierarchy. Additionally, access control policies associated with roles can change over time based on changes to factors such as the computer infrastructure 8 or any other factors. Therefore, users in the computer infrastructure 8 manually create access control policies for roles, allowing roles to perform certain functions within the computer infrastructure 8, by manually setting permission operations for objects and roles.

[0097] According to various aspects of this disclosure, components of computer infrastructure 8, such as network controller 24, can create access control policies to permit roles to access resources within computer infrastructure 8 and to perform operations on those resources. Network controller 24 can create this access control policy for a role without requiring an administrator to manually configure the exact resources accessed by the role and the exact operations performed by the role on each resource. Alternatively, a user with elevated privileges, such as an administrator of computer infrastructure 8 or a specific domain, cluster, tenant, etc., within computer infrastructure 8, can send a request to network controller 24 to create an access control policy to permit the role to perform one or more functions within computer infrastructure 8.

[0098] Network controller 24 can determine one or more operations performed on one or more resources in computer infrastructure 8 to perform one or more specified functions, and can generate access control policies for roles to permit associated roles to perform one or more operations on those resources. Therefore, network controller 24 can generate access control policies for roles to specify permitted operations performed by roles on one or more resources in computer infrastructure 8.

[0099] exist Figure 1In the example, an administrator can request the creation of an access control policy for a role to allow the role to perform one or more functions within the computer infrastructure 8 by providing instructions to network controller 24 regarding one or more functions to be performed by the role. The administrator can specify one or more functions to be performed by the role by invoking functions provided by, for example, an application programming interface (API) provided by network controller 24. For example, the administrator can provide user input at a user interface device operationally coupled to network controller 24 to specify one or more functions to be performed by the role, or the administrator can create scripts that invoke functions provided with the API, authorizing the role to make those invocations.

[0100] In some examples, administrators can also specify roles associated with access control policies. By specifying roles associated with access control policies, access control policies, generated at least in part based on one or more functions specified by the administrator, can limit the operations permitted to be performed by users and / or applications assigned to roles within the computer infrastructure.

[0101] In some examples, the administrator can also specify the scope of the access control policy created. The scope can indicate the portion of computer infrastructure 8 to which the access control policy applies. In examples where computer infrastructure 8 hosts and / or otherwise includes clusters, the administrator can specify whether a role is associated with a specific namespace so that the role's access control policy applies to that specific namespace, or the administrator can specify whether a role is associated with a specific cluster so that the role's access control policy applies to the specific clusters hosted and / or otherwise included in computer infrastructure 8.

[0102] In response to receiving an instruction for one or more functions, network controller 24 may execute those functions. In some examples, if the instruction for one or more functions includes or otherwise specifies one or more functions of the API provided by network controller 24, then network controller 24 may execute one or more specified functions of the API. Network controller 24 may be used to control components of computer infrastructure 8, such as data center 10, chassis switch 18, TOR switch 16, server 12, container platform (e.g., container platform 19), container pool (e.g., container pool 22), virtual router (e.g., virtual router 21), container network interface (e.g., CNI 17), hypervisors, policies, applications, services, etc., to execute the functions of the API.

[0103] Because network controller 24 performs API functions, it can record actions performed on objects and resources in computer infrastructure 8 as a result of function execution, creating a time log recording the sequence of actions in computer infrastructure 8. Network controller 24 is capable of recording before function execution. Because network controller 24 performs API call functions, it can record information such as the resources operated on by the API call, the operations performed on the resources (e.g., CRUD operations), the timestamps of each operation performed on the resources, the namespace of the operated resources, etc. In some examples, network controller 24 can record information as events, where each event in the audit log is associated with the operated resource and may include an indication of the resource, one or more operations performed on the resource, the timestamps of one or more operations performed on the resource, the namespace of the resource, etc.

[0104] Network controller 24 can filter and / or parse audit logs to extract information such as the resources operated on by the functions and the operations performed on those resources. In some examples, network controller 24 can filter audit logs based on timestamps associated with events recorded in the audit logs. For example, network controller 24 can determine when network controller 24 started executing a function and when network controller 24 finished executing the function. Therefore, network controller 24 can filter the audit logs for events whose timestamps occur between the time network controller 24 started executing the function and the time network controller 24 finished executing the function.

[0105] In some examples, network controller 24 can filter audit logs based on the namespace associated with the events recorded in the audit logs. If the administrator specifies that access control policies are applied to specific namespaces, network controller 24 can filter audit logs for events associated with specific namespaces.

[0106] Network controller 24 can parse audit logs, including filtered audit logs, to extract information that network controller 24 can use to generate access control policies. Specifically, network controller 24 can extract one or more resources recorded in the audit logs as accessed and can extract one or more associated operations performed on each of these resources. Therefore, network controller 24 can generate access control policies that specify one or more associated operations recorded in the access logs as actions to be performed on each of the one or more resources recorded in the audit logs as accessed.

[0107] Figure 2This is a block diagram illustrating an example of a cloud-native SDN architecture for a cloud-native network according to the technology of this disclosure. The SDN architecture 200 is shown by abstracting the underlying connections between the various components. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A-230N (“configuration nodes” or “config nodes” and collectively referred to as “configuration node 230”) and control nodes 232A-232K (collectively referred to as “control node 232”). Configuration node 230 and control node 232 can be respectively represented as… Figure 1 An exemplary implementation of configuration node 30 and control node 32 is shown. Although shown as separate from server 12, configuration node 230 and control node 232 can be executed as one or more workloads on server 12.

[0108] Configuration node 230 provides a northbound Representational State Transfer (REST) ​​interface to support intent-driven configuration of the SDN architecture 200. Exemplary platforms and applications used to push intents to configuration node 230 include virtual machine orchestrator 240 (e.g., OpenStack), container orchestrator 242 (e.g., Kubernetes), user interface 244, or one or more other applications 246. In some examples, SDN architecture 200 uses Kubernetes as its underlying platform.

[0109] The SDN architecture 200 is divided into a configuration plane, a control plane, a data plane, and an optional telemetry (or analytics) plane. The configuration plane is implemented using horizontally scalable configuration nodes 230, the control plane using horizontally scalable control nodes 232, and the data plane using compute nodes. An optional telemetry plane can be implemented using telemetry nodes 260.

[0110] At a higher level, configuration node 230 uses configuration storage 224 to manage the configuration resource state of SDN architecture 200. Typically, a configuration resource (or more simply a "resource") is a named object schema that includes data and / or methods describing a custom resource and defines an application programming interface (API) for creating and manipulating data via an API server. The type is the name of the object schema. Configuration resources can include Kubernetes native resources such as container pools, ingresses, configuration graphs, services, roles, namespaces, nodes, network policies, or load balancers. Configuration resources also include custom resources used to extend the Kubernetes platform by defining APIs not available in the default installation of the Kubernetes platform. In the example of SDN architecture 200, custom resources can describe the physical infrastructure, virtual infrastructure, configuration, and / or other resources of SDN architecture 200. Various custom resources can be instantiated as part of configuring and operating SDN architecture 200. Instantiated resources (whether native or custom) can be referred to as resource objects or instances, that is, permanent entities in SDN architecture 200 representing the intent (desired state) and condition (actual state) of SDN architecture 200. Configuration node 230 provides an aggregate API for performing operations (i.e., create, read, update, and delete) on configuration resources of SDN architecture 200 in configuration storage 224. Load balancer 226 represents one or more load balancing objects that load balance configuration requests across configuration nodes 230. Configuration storage 224 can represent one or more etcd databases. Configuration node 230 can be implemented using Nginx.

[0111] SDN Architecture 200 provides networking for OpenStack and Kubernetes. OpenStack uses a plug-in architecture to support networking. The virtual machine orchestrator 240, namely the OpenStack, OpenStack Networking plug-in driver, transforms OpenStack configuration objects into SDN Architecture 200 configuration objects (resources). Compute nodes run OpenStack nova to build virtual machines.

[0112] The container orchestrator 242, i.e., Kubernetes, is used as the Kubernetes CNI. As mentioned above, it can support Kubernetes native resources (container pools, services, ingress, external load balancers, etc.), and the SDN architecture 200 can support custom Kubernetes resources for advanced networking and security.

[0113] Configuration node 230 provides REST monitoring to control node 232 to monitor changes in configuration resources. Within the computing infrastructure, control node 232 influences these changes. Control node 232 receives configuration resource data from configuration node 230 by monitoring resources and constructs a complete configuration graph. A given control node in control node 232 consumes configuration resource data associated with that control node and transmits it via virtual router 21 (i.e., virtual router proxy). Figure 1 The control interface 254 (not shown) on the control plane distributes the required configuration to the compute nodes (server 12). Any compute node may receive only a portion of the graph, as needed for processing. The control interface 254 may be XMPP. The number of configuration nodes 230 and control nodes 232 may be a function of the number of supported clusters. To support high availability, the configuration plane may include 2N+1 configuration nodes 230 and 2N control nodes 232.

[0114] Control node 232 distributes routes among compute nodes. Control nodes 232 use the Interior Border Gateway Protocol (iBGP) to exchange routes among themselves, and can peer with any external BGP-supported gateway or other router. Control node 232 can use route reflectors.

[0115] Container pool 250 and virtual machine 252 are examples of workloads that can be deployed to compute nodes by virtual machine orchestrator 240 or container orchestrator 242 and interconnected by SDN architecture 200 using one or more virtual networks.

[0116] According to various aspects of this disclosure, network controller 24 can create access control policies that permit roles to access resources and perform operations on resources of configuration architecture 200. As described above, resources may include configuration resources such as Kubernetes native resources (e.g., container pools, ingress, configuration graphs, services, roles, namespaces, nodes, network policies, or load balancers). Configuration resources also include custom resources that describe the physical infrastructure, virtual infrastructure, configuration, and / or other resources of SDN architecture 200. In some examples, resources may also include any other resources of SDN architecture 200.

[0117] An administrator may request the creation of access control policies for a role to enable one or more functions of the network controller 24 by providing instructions to the network controller 24 via user interface 244 and / or application 246 to allow the role to perform one or more functions. For example, an administrator may provide user input at a user interface device operationally coupled to the network controller 24 to specify one or more actions that the role is permitted to perform.

[0118] Administrators can send requests to the configuration node via user interface 244 to generate access policies for roles. This allows them to specify one or more actions that the permitted roles can perform on resources of the SDN architecture 200, such as creating, reading, updating, and deleting, by specifying the functionality of the aggregate API provided by configuration node 230. For example, an administrator can interact with configuration node 230 via user interface 244 to invoke the functionality of the aggregate API that the permitted roles should perform. In another example, an administrator can create a script that invokes the functionality of the aggregate API that the permitted roles should perform and communicate the script to configuration node 230. In this way, the administrator can specify one or more functions that the permitted roles can perform.

[0119] Administrators can also specify roles associated with access control policies. In the example where network controller 24 is part of or associated with a Kubernetes cluster, the administrator can specify the role associated with the access control policy as a ClusterRole role or a role associated with a specific namespace. If the administrator specifies the role as a ClusterRole role, the access control policy for that role can be applied to the Kubernetes cluster that is part of or associated with network controller 24. If the administrator specifies the role as a role associated with a specific namespace, the role associated with the access control policy can be applied to the specific namespace within the Kubernetes cluster.

[0120] The configuration node 230 of network controller 24 can receive requests and execute one or more specified functions of the aggregate API. As described above, in some examples, the aggregate API can be invoked to perform operations on resources of SDN architecture 200, such as Kubernetes native resources (e.g., container pools, ingress, configuration graphs, services, roles, namespaces, nodes, network policies, or load balancers), as well as physical infrastructure, virtual infrastructure, configured custom resources, and / or other resources that describe SDN architecture 200.

[0121] Since configuration node 230 executes the functionality of the aggregate API, it can also perform operations (e.g., CRUD operations) on one or more resources that are not explicitly specified by invoking the functionality of the aggregate API. Because resources are hierarchical, resources may depend on one or more other sub-resources. Although the functionality of the aggregate API can indicate the operation to be performed on a resource, the functionality may not necessarily indicate one or more sub-resources that the indicated resource depends on. Therefore, in order to perform one or more operations on a resource specified by the functionality of the aggregate API, configuration node 230 can determine one or more sub-resources that the resource depends on based on the resource indicated by the functionality, and can perform one or more operations on each of the one or more sub-resources, i.e., throughout any resource described in this disclosure, to successfully execute the specified functionality.

[0122] Since configuration node 230 performs one or more functions, network controller 24 can use telemetry node 260 to record actions performed on resources within SDN architecture 200 as a result of executing one or more functions of an aggregated API specified by an administrator, to create a time log of the actions performed on the resources. Configuration node 230 can record and / or audit operations performed on resources by using an audit policy file that causes telemetry node 260 to begin recording. An example of a portion of the audit policy file is shown below:

[0123] apiVersion:audit.k8s.io / v1

[0124] kind:Policy

[0125] rules:

[0126] -level:Request

[0127] resources:net

[0128] -group:"core.contrail.juniper."

[0129] Because node 230 is configured to perform API calls, telemetry node 260 can log information such as the resources operated on by the API call functionality, the operations performed on the resources (e.g., CRUD operations), the timestamps of each operation performed on the resources, and the namespace of the operated resources. In some examples, telemetry node 260 can log information as events, where each event in the audit log is associated with the operated resource and may include indications of the resource, one or more operations performed on the resource, the timestamps of one or more operations performed on the resource, and the namespace of the resource. An example of a portion of the audit log is shown below:

[0130] {"kind":"Event",

[0131] "apiVersion":"audit.k8s.io / v1",

[0132] "level":"Request",

[0133] "auditID":"61005be0-02cb-4286-acb6-ade45f99ecc3",

[0134] "stage":"RequestReceived",

[0135] "requestURI":" / apis / core.contrail.juniper.net / v1alpha1 / virtualnetworks?watch=1\u0026resourceVersion=5086",

[0136] "verb":"watch",

[0137] "user":{"username":"system:serviceaccount:contrail:contrail-serviceaccount",

[0138] "uid":"cd07eea1-e18a-48cf-a4e6-4f3800479eab","groups":["system:serviceaccounts",

[0139] "system:serviceaccounts:contrail","system:authenticated"]},

[0140] "sourceIPs":["10.88.0.1"],

[0141] "userAgent":"restclient-cpp / 0.5.2",

[0142] "objectRef":{"resource":"virtualnetworks","apiGroup":"core.contrail.juniper.net","apiVersion":"v1alpha1"},

[0143] "requestReceivedTimestamp":"2021-09-13T18:36:53.761497Z",

[0144] "stageTimestamp":"2021-09-13T18:36:53.761497Z",

[0145] "annotations":{"authentication.k8s.io / legacy-token":"system:serviceaccount:contrail:contrail-serviceaccount"}}

[0146] This example demonstrates that telemetry node 260 can record information such as resources (e.g., "virtual networks"), operations performed on resources (e.g., "monitoring"), timestamps of events, and other information. As shown in the example, operations on resources are referred to as Kubernetes verbs, such as get, list, monitor, update, create, and patch.

[0147] Configuration node 230 can filter and / or parse audit logs to extract information such as the resource operated on by the API call (e.g., type) and the operation performed on the resource (e.g., verb). In some examples, configuration node 230 can filter audit logs based on timestamps associated with events recorded in the audit logs. For example, configuration node 230 can determine when configuration node 230 started executing the API call and when configuration node 230 finished executing the API call. Therefore, configuration node 230 can filter the audit logs for events whose timestamps occur between the time configuration node 230 started executing the API call and the time configuration node 230 finished executing the API call.

[0148] In some examples, configuration node 230 can filter audit logs based on the namespace associated with the events recorded in the audit logs. In some examples, if the administrator specifies that access control policies are applied to specific namespaces of the cluster, configuration node 230 can filter audit logs based on events associated with a specific namespace, such as the namespace associated with each event recorded in the audit logs.

[0149] Configuration node 230 can parse audit logs, including filtered audit logs, to extract information that network controller 24 can use to generate access control policies. Specifically, configuration node 230 can extract one or more resources recorded in the audit logs as accessed and can extract one or more actions performed on each of those resources. For example, configuration node 230 can extract resources and actions as ['kind', 'verb'] pairs referenced by type. An example of extracting resources and actions is as follows:

[0150] {

[0151] "globalsystemconfigs":["watch","update"],

[0152] "routetargets":["create","get","watch"],

[0153] "routinginstances":["create","patch","list","update","watch"],

[0154] "subnets":["get","watch"],

[0155] "virtualmachineinterfaces":["watch","list","update"],

[0156] "virtualnetworks":["create","patch","update","watch"],

[0157] }

[0158] As shown in the example above, configuration node 230 can extract the "globalsystemconfigs" resource using the "monitor" and "update" operations performed on the resource; extract the "routetargets" resource using the "create", "get", and "monitor" operations performed on the resource; extract the "routinginstances" resource using the "create", "patchase", "list", "update", and "monitor" operations performed on the resource; extract the "subnets" resource using the "get" and "monitor" operations performed on the resource; extract the "virtualmachineinterfaces" resource using the "monitor", "list", and "update" operations performed on the resource; and extract the "virtualnetworks" resource using the "create", "patchase", "update", and "monitor" operations performed on the resource.

[0159] Therefore, configuring node 230 can generate access control policies that specify one or more associated operations recorded in the access logs and performed on the resource for each of one or more resources accessed as recorded in the audit logs, as permitted operations for that resource. An example of such an access control policy is as follows:

[0160] kubectl describeClusterRoledev

[0161] Name:dev

[0162] Labels: <none>

[0163] Annotations: <none>

[0164] Policy Rule:

[0165] Resources Non-Resource URLs Resource Names Verbs

[0166] ----------------routetargets.core.contrail.juniper.net[][][create]

[0167] routinginstances.core.contrail.juniper.net[][][create]

[0168] virtualnetworks.core.contrail.juniper.net[][][create]

[0169] subnets.core.contrail.juniper.net[][][get]

[0170] In the example above, the access control policy for the ClusterRole role named "dev" enables the role to create routetarget resources, routinginstance resources, virtualnetwork resources, and subnets resources.

[0171] Network controller 24 can use generated access control policies to ensure that users and services with associated roles cannot access resources by violating the access control policies associated with those roles. For example, if a user has a "dev" role associated with the exemplary access control policy detailed above, and if the user invokes a function in the API to perform an operation on the "subnets" resource, network controller 24 can determine whether the operation on the resource is permitted or whether the operation violates the access policy of the "dev" role. For example, because the exemplary access control policy detailed above restricts the "dev" role from performing an "acquire" operation on the "subnets" resource, network controller 24 might disallow the "create" operation on the "subnets" resource if the user invokes a function in the API to perform a "create" operation on the "subnets" resource. Thus, network controller 24 can constantly monitor for potential access control policy violations by comparing the functions invoked by the user with the access control policy of that user's role using generated access control policies.

[0172] In some examples, administrators can use the techniques described above to update access control policies for roles. For instance, an administrator can update a role's access control policy to allow the role to execute one or more additional functions in network controller 24 by specifying one or more additional functions of the aggregate API to configuration node 230. Configuration node 230 can execute one or more additional functions specified by the administrator and can log the execution of such additional functions in the audit log. Configuration node 230 can filter and / or parse the audit log to extract information such as the resources operated on by executing the additional functions and the operations performed on those resources. Therefore, configuration node 230 can update the role's access control policy as one or more operations allowed to be performed on the resource for each of the one or more resources accessed as recorded in the audit log, including one or more associated operations performed on the resource as recorded in the access log.

[0173] In some examples, network controller 24 can determine whether a generated access control policy violates a configured access control policy in network controller 24. That is, by comparing a role's access control policy with the role's configured access control policies (which may be stored in one or more configuration stores), network controller 24 may violate a role's generated access control policy. For example, if network controller 24 determines that a role's generated access control policy permits the role to perform operations on resources not permitted by the role's configured access control policy, then network controller 24 can determine that the generated access control policy is invalid. In another example, if network controller 24 determines that a role's generated access control policy permits the role to perform operations on resources not permitted by the role's configured access control policy, then network controller 24 can update the role's configured access control policy to permit the role to perform operations on resources permitted by the generated access control policy.

[0174] In some examples, network controller 24 may provide visualization of access control policies by outputting a graphical representation of the access control policies, for display at, for example, user interface 244. Providing this visualization of access control policies allows users (e.g., administrators) to view the details of the access control policies to determine whether network controller 24 has correctly generated the access control policies for roles.

[0175] Figure 3 This is a block diagram showing, and further in detail, another view of the components of the SDN architecture 200 according to the technology of this disclosure. Configuration node 230, control node 232, and user interface 244 are shown as having their respective component microservices to implement network controller 24 and SDN architecture 200 as a cloud-native SDN architecture. Each component microservice can be deployed to compute nodes.

[0176] Figure 3 A single cluster is shown, divided into network controller 24, user interface 244, compute nodes (server 12), and telemetry features 260. Configuration node 230, together with control node 232, constitutes network controller 24.

[0177] Node 230 can include the component microservice API server 300 (or "Kubernetes API server 300" - the corresponding controller 406). Figure 3 The configuration node 230 utilizes the custom API server 301 to extend the interface of the API server 300, forming an aggregation layer to support the data model of the SDN architecture 200. As mentioned above, the configuration intent of the SDN architecture 200 can be a custom resource controller 302, an SDN controller manager 303 (sometimes referred to as "kube-manager" or "SDN kube-manager", where the orchestration platform for the network controller 24 is Kubernetes), and an access control manager 305. The Contrail-kube-manager is an example of the SDN controller manager 303.

[0178] Control node 232 may include component microservice control 320 and core DNS 322. (See above reference.) Figure 2 As described, Control 320 performs configuration allocation and route learning and allocation.

[0179] Server 12 represents the compute nodes. Each compute node includes a virtual router agent 316 and a virtual router forwarding component (vRouter) 318. Either the virtual router agent 316 or the vRouter 318 can be a component microservice, or both can be component microservices. Typically, the virtual router agent 316 performs control-related functions. The virtual router agent 316 receives configuration data from the control node 232 and converts the configuration data into forwarding information for the vRouter 318. The virtual router agent 316 can also perform firewall rule processing, configure the flow of the vRouter 318, and interface with orchestration plugins (CNI for Kubernetes and Nova plugin for OpenStack). When a workload (container pool or VM) is called up on a compute node, the virtual router agent 316 generates a route, and the virtual router 318 exchanges the route with the control node 232 to distribute it to other compute nodes (the control node 232 uses BGP to distribute routes between control nodes 232). When the workload is terminated, the virtual router agent 316 also drops the route. vRouter 318 can support one or more forwarding modes, such as kernel mode, DPDK, smart NIC offloading, etc. In some examples of container architectures or virtual machine workloads, compute nodes can be Kubernetes worker / slave nodes or OpenStack nova compute nodes, depending on the specific orchestrator used.

[0180] One or more optional telemetry nodes 260 provide metrics, alerts, logging, and stream analytics. Telemetry for the SDN architecture 200 utilizes cloud-native monitoring services such as Prometheus, Elastic, Fluentd, Kinabalu stack (EFK), and Influx TSDB. SDN architecture component microservices, including configuration node 230, control node 232, compute nodes, user interface 244, and analytics nodes (not shown), can generate telemetry data. The services of telemetry node 260 can use this telemetry data. Telemetry node 260 can expose REST endpoints for users and support insights and event correlation.

[0181] Optionally, the user interface 244 includes a network user interface (UI) 306 and a UI backend 308 service. Typically, the user interface 244 provides configuration, monitoring, visualization, security, and troubleshooting for SDN architecture components.

[0182] Each of the telemetry node 260, user interface 244, configuration node 230, control node 232, and server 12 / compute node can be considered a node of the SDN architecture 200, in that each of these nodes is an entity that implements the functionality of the configuration plane, control plane, or data plane, or UI and telemetry node. The node size is configured during the "build" process, and the SDN architecture 200 uses an orchestration system operator such as the Kubernetes operator to support the automatic scaling of the SDN architecture 200 nodes.

[0183] According to various aspects of this disclosure, the access control manager 305 is configured to generate access control policies for roles to perform one or more functions in the network controller 24. An administrator can request the creation of access control policies for roles to authorize roles to perform one or more functions in the network controller 24 by providing instructions to the network controller 24 via the network UI 306 or backend 308. An administrator can also send a request to the configuration node 230 to generate access policies for roles via the network UI 306 or backend 308 to specify one or more functions that roles are authorized to perform by performing CRUD operations on resources of the SDN architecture 200 by specifying the functionality of the aggregated APIs provided by the API server 300 and / or the custom API server 301.

[0184] API server 300 and / or custom API server 301 can receive requests and configure node 230 to execute one or more specified functions of the aggregated API. Since node 230 is configured to execute one or more functions, access control manager 305 can use telemetry node 260 to record actions performed on resources within the SDN architecture 200 as a result of executing one or more functions of the aggregated API specified by the administrator, to create a time log of the actions performed on the resources. Access control manager 305 can record and / or audit the operations performed on the resources by using an audit policy file that causes telemetry node 260 to begin recording.

[0185] Because the configuration node 230 performs the functionality of the aggregate API, the telemetry node 260 can log information such as the resources operated on by the API call, the operations performed on the resources (e.g., CRUD operations), the timestamps of each operation performed on the resources, and the namespace of the operated resources. In some examples, the telemetry node 260 can log information as events, where each event in the audit log is associated with the operated resource and may include indications of the resource, one or more operations performed on the resource, the timestamps of one or more operations performed on the resource, and the namespace of the resource.

[0186] Access control manager 305 can filter and / or parse audit logs to extract information such as the resources operated on by API calls and the operations performed on those resources (e.g., verbs). In some examples, access control manager 305 can filter audit logs based on timestamps associated with events recorded in the audit logs. In some examples, configuration node 230 can filter audit logs based on namespaces associated with events recorded in the audit logs. If the administrator specifies that access control policies are applied to specific namespaces of the cluster, configuration node 230 can filter audit logs for events associated with those specific namespaces.

[0187] Configuration node 230 can parse audit logs, including filtered audit logs, to extract information used by access control manager 305 to generate access control policies. Specifically, access control policy 305 can extract one or more resources recorded in the audit logs as accessed and can extract one or more associated operations performed on each of the one or more resources, thereby associating each resource with one or more operations. For example, access control manager 305 can extract resources and operations as ['kind', 'verb'] pairs. An example of extracting resources and operations is as follows:

[0188] Therefore, the access control manager 305 can generate access control policies that specify one or more actions recorded in the access log as being performed on the resource for each of one or more resources accessed as recorded in the audit log. The network controller 24 can use the generated access control policies to ensure that users and services with associated roles cannot access resources by violating the access control policies associated with those roles.

[0189] In some examples, administrators can use the techniques described above to update access control policies for roles. For instance, an administrator can update a role's access control policy to allow the role to execute one or more additional functions in network controller 24 by specifying one or more additional functions of the aggregate API to configuration node 230. Configuration node 230 can execute one or more additional functions specified by the administrator and can log the execution of such additional functions in the audit log. Access control manager 305 can filter and / or parse the audit log to extract information such as the resources operated on by executing the additional functions and the operations performed on those resources. Therefore, access control manager 305 can update the role's access control policy to include, for each of the one or more resources recorded in the audit log as accessed, one or more associated operations performed on the resource as permitted by the policy.

[0190] In some examples, access control manager 305 can determine whether a generated access control policy violates a configured access control policy in network controller 24. That is, by comparing a role's access control policy with the role's configured access control policies (which may be stored in one or more configuration stores), access control manager 305 may violate a role's generated access control policy. For example, if access control manager 305 determines that a role's generated access control policy permits the role to perform operations on resources not permitted by the role's configured access control policy, then access control manager 305 can determine that the generated access control policy is invalid. In another example, if access control manager 305 determines that a role's generated access control policy permits the role to perform operations on resources not permitted by the role's configured access control policy, then access control manager 305 can update the role's configured access control policy to permit the role to perform operations on resources permitted by the generated access control policy.

[0191] Figure 4 This is a block diagram illustrating exemplary components of an SDN architecture according to the technology disclosed herein. In this example, the SDN architecture 400 extends and utilizes a Kubernetes API server for network configuration objects to implement the user intent of that network configuration. In Kubernetes terminology, this configuration object is referred to as a custom resource, and when it exists in the SDN architecture, it is simply referred to as an object. The configuration object is primarily a user intent (e.g., virtual network, BGPaaS, network policy, service chain, etc.).

[0192] The configuration node 230 of the SDN architecture 400 can use the Kubernetes API server to configure objects. In Kubernetes terminology, this is called a custom resource.

[0193] Kubernetes provides two ways to add custom resources to the cluster:

[0194] Custom resource definitions (CRDs) are simple to create and can be done without any programming.

[0195] API aggregation requires programming, but it allows for greater control over API behavior, such as how data is stored and how API versions are converted.

[0196] An aggregated API is a subordinate API server that sits behind the main API server, acting as a proxy. This arrangement is called API Aggregation (AA). To the user, it appears simply as an extension of the Kubernetes API. Custom Resource Definitions (CRs) allow users to create new types of resources without adding another API server. Regardless of how it's installed, the new resource is called a Custom Resource (CR) to distinguish it from native Kubernetes resources (e.g., container pools). CRs are used in the initial configuration prototype. This architecture can be implemented using the API Server Builder Alpha library. API Server Builders are a collective term for libraries and tools used to build native Kubernetes aggregate extensions.

[0197] Typically, each resource in the Kubernetes API requires code to handle REST requests and manage persistent storage for objects. The main Kubernetes API server 300 (implemented via API server microservices 300A-300J) handles native resources and can also perform general processing on custom resources via CRDs. Aggregate API 402 represents an aggregation layer that extends the Kubernetes API server 300 to allow for specialized implementations of custom resources by writing and deploying custom API servers 301 (using custom API server microservices 301A-301M). The main API server 300 delegates requests for custom resources to custom API servers 301, thereby making the resource available to all its clients.

[0198] Thus, API server 300 (e.g., kube-api server) receives Kubernetes configuration objects, local objects (container pools, services), and custom resources as defined by the techniques of this disclosure. Custom resources for SDN architecture 400 may include configuration objects that implement the expected network configuration of SDN architecture 400 when the expected state of the configuration object is implemented in SDN architecture 400. Custom resources may correspond to configuration schemes that are conventionally constrained for network configuration but conform to the techniques of this disclosure, and are extended to be manipulated via aggregate API 402. Alternatively, the custom resource may be named and referred to herein as "custom resources for SDN architecture configuration." These may include virtual networks, BGP as a Service (BGPaaS), subnets, virtual routers, service instances, projects, physical interfaces, logical interfaces, nodes, network IPAMs, floating IPs, alerts, alias IPs, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances, etc. Custom resources used for SDN architecture configuration can correspond to configuration objects typically exposed by SDN controllers. However, according to the techniques described herein, configuration objects are exposed as custom resources and merged with Kubernetes native / built-in resources to support a unified intent model exposed by the aggregate API 402, i.e., through Kubernetes controllers 406A-406N and through the custom resource controller 302 (in... Figure 4 The unified intent model is implemented as a component microservice (302A-302L). The Kubernetes custom resource controller 302 is dedicated to reconciling the actual state of the computing infrastructure, including network components, into the intended state.

[0199] The aggregation layer of API server 300 sends API custom resources to its corresponding registered custom API server 301. Multiple custom API servers / custom resource controllers may exist that support different types of custom resources. Custom API server 300 processes the custom resources used for SDN architecture configuration and writes the custom resources to configuration store 304 (which may be etcd). Custom API server 300 can be a host and exposes the SDN controller identifier allocation service required by custom resource controller 302.

[0200] Custom resource controller 302 begins applying business logic to satisfy user intents configured with user intent settings. This business logic is implemented as a coordination loop. Figure 8 This is a block diagram illustrating an example of a custom controller for configuring custom resources in an SDN architecture according to the technology of this disclosure. User controller 814 may represent an exemplary instance of custom resource controller 302. Figure 8 In the example shown, a custom controller 814 can be associated with a custom resource 818. Custom resource 818 can be any custom resource used for SDN architecture configuration. Custom controller 814 can include a coordinator 816, which includes logic for executing a coordination loop, wherein custom controller 814 observes (e.g., monitors) the current state 832 of custom resource 818. In response to determining that the desired state 826 does not match the current state 832, coordinator 816 can perform an action to adjust the state of the custom resource 838 so that the current state 832 matches the desired state 836. A request can be received by API server 300 and relayed to custom API server 301 to change the current state 832 of custom resource 818 to the desired state 836.

[0201] When an API request 301 is a creation request for a custom resource, the coordinator 816 can act on a creation event for instance data of the custom resource. The coordinator 816 can create instance data for the custom resources that the requested custom resource depends on. For example, an edge node custom resource can depend on a virtual network custom resource, a virtual interface custom resource, and an IP address custom resource. In this example, when the coordinator 816 receives a creation event for an edge node custom resource, it can also create the custom resources that the edge node custom resource depends on, such as the virtual network custom resource, the virtual interface custom resource, and the IP address custom resource.

[0202] By default, the custom resource controller 302 runs in an active-passive mode and uses a master election to achieve consistency. When the controller container pool starts, it attempts to create a ConfigMap resource in Kubernetes using a specified key. If creation is successful, the container pool becomes the master and begins handling coordination requests; otherwise, the container pool blocks attempts to create an infinite loop of ConfigMaps.

[0203] The custom resource controller 302 can track the status of the custom resources it creates. For example, a virtual network (VN) creates a route instance (RI), and the route instance (RI) creates a route destination (RT). If creating the route destination fails, the status of the route instance degrades, and consequently, the status of the virtual network also degrades. Therefore, the custom resource controller 302 can output custom messages indicating the status of these custom resources to troubleshoot the problem. Figure 9 The diagram illustrates the creation, observation, and coordination of exemplary flows between custom resource types that depend on different custom resource types.

[0204] The configuration plane implemented by configuration node 230 is highly available. Configuration node 230 can be based on Kubernetes and includes a kube-api server service (e.g., API server 300) and a storage backend etcd (e.g., configuration storage 304). In practice, the aggregate API 402 implemented by configuration node 230 acts as a front end of the control plane implemented by control node 232. The primary implementation of API server 300 is designed as a horizontally scalable kube-api server by deploying more instances. As shown, several instances of API server 300 can be run to load balance API requests and processing.

[0205] Configuring a 304 configuration can be implemented as etcd. etcd is a consistent and highly available key-value store used as a backup storage for cluster data in Kubernetes.

[0206] exist Figure 4 In the example, each of the servers 12 in the SDN architecture 400 includes an orchestration agent 420 and a containerized (or "cloud-native") routing protocol daemon (cRPD) 324. These components of the SDN architecture 400 are described in further detail below.

[0207] SDN Controller Manager 303 acts as an interface between Kubernetes core resources (services, namespaces, container pools, network policies, network attachment definitions) and extended SDN architecture resources (virtual networks, routing instances, etc.). SDN Controller Manager 303 monitors changes to the Kubernetes API on custom resources configured in the Kubernetes core network and SDN architecture, and thus can perform CRUD operations on the relevant resources.

[0208] In some examples, SDN Controller Manager 303 is a collective term for one or more Kubernetes custom controllers. In some examples, in a single or multiple cluster deployment, SDN Controller Manager 303 can run on the Kubernetes clusters it manages.

[0209] SDN Controller Manager 303 listens for the following Kubernetes objects to perform creation, deletion, and update events:

[0210] ·Container pool

[0211] ·Serve

[0212] Node port

[0213] ·Entrance

[0214] • Endpoint

[0215] Namespace

[0216] ·deploy

[0217] • Network strategy.

[0218] When these events are generated, the SDN controller manager 303 creates an appropriate SDN architecture object, which is then defined as a custom resource for the SDN architecture configuration. In response to detecting an event on an instance of the custom resource, whether instantiated by the SDN controller manager 303 and / or via the custom API server 301, the control node 232 obtains configuration data about the instance of the custom resource and configures the corresponding instance of the configuration object in the SDN architecture 400.

[0219] For example, the SDN controller manager 303 monitors container pool creation events and, accordingly, can create the following SDN architecture objects: virtual machines (workloads / container pools), virtual machine interfaces (virtual network interfaces), and instance IPs (IP addresses). Then, in this case, the control node 232 can instantiate the SDN architecture objects in the selected compute node.

[0220] According to various aspects of this disclosure, an administrator may request the creation of access control policies for a role to permit the role to perform one or more functions within the SDN architecture by providing instructions on the permitted functions performed by the role. The administrator may send a request to configuration node 230 to generate access policies for a role to specify one or more actions permitted for the role to perform on resources of the SDN architecture 200 by specifying the functionality of the aggregate API 402 provided by API server 300 and / or custom API server 301.

[0221] API server 300 and / or custom API server 301 can receive requests, and configuration node 230 can perform one or more specified functions on the aggregated API. As configuration node 230 performs these functions, it can record the actions performed on resources within the SDN architecture 400 as a result of executing one or more functions of the aggregated API specified by the administrator, creating a time log of the actions performed on the resources. Configuration node 230 can record and / or audit the operations performed on resources by using an audit policy file that causes telemetry nodes to begin recording.

[0222] Because configuration node 230 performs the function of the aggregate API, the telemetry node can log information such as the resources operated on by the function that performs the API call, the operations performed on the resources (e.g., CRUD operations), the timestamps of each operation performed on the resources, and the namespace of the operated resources. In some examples, the telemetry node can log information as events, where each event in the audit log is associated with the operated resource and may include indications of the resource, one or more operations performed on the resource, the timestamps of one or more operations performed on the resource, and the namespace of the resource.

[0223] Configuration node 230 can filter and / or parse audit logs to extract information such as the resources operated on by the API calls and the operations performed on those resources (e.g., verbs). In some examples, configuration node 230 can filter audit logs based on the timestamps associated with the events recorded in the audit logs. In some examples, configuration node 230 can filter audit logs based on the namespaces associated with the events recorded in the audit logs. If the administrator specifies that access control policies are applied to specific namespaces of the cluster, configuration node 230 can filter audit logs for events associated with specific namespaces.

[0224] Configuration node 230 can parse audit logs, including filtered audit logs, to extract information for generating access control policies. Specifically, configuration node 230 can extract one or more resources recorded in the audit logs as accessed and can extract one or more associated operations performed on each of these resources, thereby associating each resource with one or more operations. For example, configuration node 230 can extract resources and operations as ['kind', 'verb'] pairs. An example of extracting resources and operations is as follows:

[0225] Therefore, configuration node 230 can generate access control policies to specify one or more actions recorded as performed on a resource in the access log for each of one or more resources accessed as recorded in the audit log. Configuration node 230 can use the generated access control policies to ensure that users and services associated with roles cannot access resources by violating the access control policies associated with those roles.

[0226] In some examples, the configuration node can determine whether a generated access control policy violates a configured access control policy in the configuration node. That is, by comparing a role's access control policy with the role's configured access control policies (which may be stored in one or more configuration stores 304), configuration node 230 may violate a role's generated access control policy. For example, if configuration node 230 determines that a role's generated access control policy permits the role to perform operations on resources that are not permitted by the role's configured access control policy, then configuration node 230 can determine that the generated access control policy is invalid. In another example, if configuration node 230 determines that a role's generated access control policy permits the role to perform operations on resources that are not permitted by the role's configured access control policy, then configuration node 230 can update the role's configured access control policy to permit the role to perform operations on resources permitted by the generated access control policy.

[0227] Figure 5 This is a block diagram of an exemplary computing device according to the technology described in this disclosure. Figure 5 The computing device 500 in the example can represent an actual server or a virtual server and can represent an exemplary example of any server 12 and can be referred to as a compute node, master / slave node, or host. In this example, the computing device 500 includes a bus 542 coupled to hardware components in the hardware environment of the computing device 500. The bus 542 is coupled to a network interface card (NIC) 530, a storage disk 546, and one or more microprocessors 510 (hereinafter referred to as "microprocessor 510"). The NIC 530 may support SR-IOV. In some cases, the front-side bus may be coupled to the microprocessor 510 and a memory device 544. In some examples, the bus 542 may be coupled to the memory device 544, the microprocessor 510, and the NIC 530. The bus 542 may represent a Peripheral Component Interface (PCI) Faster (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to the bus 542. In some examples, components coupled to the bus 542 control DMA transfers between components coupled to the bus 542.

[0228] Microprocessor 510 may include one or more processors, each of which includes an independent execution unit that executes instructions conforming to an instruction set architecture and stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip microprocessor).

[0229] Disk 546 represents a computer-readable storage medium, including volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technique for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, disk storage, or other magnetic storage devices, or any other medium capable of storing desired information and accessible to the microprocessor 510.

[0230] Main memory 544 includes one or more computer-readable storage media, which may include various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), random access memory (RAM) of flash memory, or any other form of fixed or removable storage medium capable of carrying or storing required program code and program data in the form of instructions or data structures that are accessible to a computer. Main memory 544 provides a physical address space consisting of addressable memory locations.

[0231] Network interface card (NIC) 530 includes one or more interfaces 532 configured to exchange packets using links of the underlying physical network. Interface 532 may include a port interface card having one or more network ports. NIC 530 may also include on-card memory, for example, for storing packet data. Direct memory access transfers between NIC 530 and other devices coupled to bus 542 can be read from the NIC memory, or direct memory access transfers between NIC 530 and other devices coupled to bus 542 can be written to the NIC memory.

[0232] Memory 544, NIC 530, storage disk 546, and microprocessor 510 provide an operating environment for the software stack including an operating system kernel 580 executing in kernel space. For example, kernel 580 may represent Linux, Berkeley Software Distribution (BSD), another Unix variant's kernel, or a Windows server operating system kernel commercially available from Microsoft. In some examples, the operating system may run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor can encompass a Virtual Machine Manager (VMM). The operating system including kernel 580 provides an execution environment for one or more processes in user space 545.

[0233] Kernel 580 includes a physical driver 525 that uses network interface card 530. Network interface card 530 can also implement SR-IOV to enable communication with containers such as 529A or one or more virtual machines (…). Figure 5 Physical network functions (I / O) are shared between one or more virtual execution elements (not shown in the diagram). Shared virtual devices, such as virtual functions, can provide dedicated resources so that each virtual execution element can access the dedicated resources of NIC 530, thus making NIC 530 appear as a dedicated NIC for each virtual execution element. Virtual functions can represent lightweight PCIe functions that share physical resources with physical drivers 525 and other virtual functions. For an SR-IOV-enabled NIC 530, NIC 530 can have thousands of available virtual functions compliant with the SR-IOV standard; however, for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0234] Computing device 500 can be coupled to a physical network switching infrastructure, which includes an overlay network that extends the switching infrastructure from physical switches to software or "virtual" routers (including virtual router 506) coupled to physical servers. The virtual router can be a physical server (e.g., Figure 1 The virtual router, through processes or threads, or combinations thereof, executed by server 12, dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints. In one example, each virtual router implements a virtual network using an overlay network, which provides the ability to decouple the virtual address of an endpoint from its physical address (e.g., IP address) on the server executing the endpoint. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and across virtual networks above a physical network. As used herein, the term "virtual router" may encompass Open vSwitch (OVS), OVS bridges, Linux bridges, Docker bridges, or other devices and / or software residing on a host device and performing the switching, bridging, or routing of packets between virtual network endpoints in one or more virtual networks, wherein the virtual network endpoints are hosted by one or more servers 12. Figure 5 In the exemplary computing device 500, the virtual machine router 506 operates as a DPDK-based virtual router in user space; however, the virtual router 506 can be implemented in various ways within a hypervisor, host operating system, host application, or virtual machine.

[0235] Virtual Router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used in Kubernetes deployments of Container Pool 502. Virtual Router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) on virtual networks. Virtual Router 506 can perform network services such as application security policies, NAT, multicast, mirroring, and load balancing.

[0236] Virtual router 506 can run as a kernel module or as a user-space DPDK process (here, virtual router 506 is shown located in user space 545). Virtual router agent 514 can also run in user space. In the exemplary computing device 500, virtual router 506 runs as a DPDK-based virtual router within user space; however, virtual router 506 can run within a hypervisor, host operating system, host application, or virtual machine through various implementations. Virtual router agent 514 uses a channel to connect to network controller 24, which is used to download configuration and forwarding information. Virtual router agent 514 programs this forwarding state into a virtual router data (or "forwarding") plane represented by virtual router 506. Virtual router 506 and virtual router agent 514 can be processes. Virtual router 506 and virtual router agent 514 are containerized / cloud-native.

[0237] Virtual Router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used in Kubernetes deployments of Container Pool 502. Virtual Router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) on virtual networks. Virtual Router 506 can perform network services such as application security policies, NAT, multicast, mirroring, and load balancing.

[0238] The virtual router 506 can be multi-threaded and run on one or more processor cores. The virtual router 506 can include multiple queues. The virtual router 506 can implement packet processing pipelines. Depending on the operations applied to the packets, the virtual router agent 514 stitches the pipeline together in the simplest to the most complex manner. The virtual router 506 can maintain multiple instances of the forwarding base. The virtual router 506 can use RCU (Read Copy Update) locks to access and update tables.

[0239] To send packets to other compute nodes or switches, virtual router 506 uses workloads injected into VMs or container pools 502. Virtual router 506 has multiple virtual network interfaces (e.g., Vifs). These interfaces may include a kernel interface vhost0 for exchanging packets with the host operating system, and an interface pkt0 for exchanging packets with virtual router agent 514 to obtain forwarding status from the network controller and send exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 532. Other virtual network interfaces of virtual router 506 are used to exchange packets with workloads.

[0240] In a kernel-based deployment of the virtual router 506 (not shown), the virtual router 506 is installed as a kernel module within the operating system. The virtual router 506 registers itself via the TCP / IP stack to receive packets from any desired operating system interface. Interfaces can be bound, physicalized, de-splittered (for VMs), veth-based (for containers), etc. In this mode, the virtual router 506 relies on the operating system to send packets to and receive packets from different interfaces. For example, the operating system can expose a de-splitter interface supported by the vhost network driver to communicate with VMs. Once the virtual router 506 registers packets from that de-splitter interface, the TCP / IP stack sends all packets to the virtual router. The virtual router 506 sends packets via the operating system. Additionally, the operating system handles NIC queues (physical or virtual). Packet processing can operate in interrupt mode, which generates interrupts and may result in frequent context switching. At high packet rates, the total overhead of frequent interrupts and context switching can overwhelm the operating system and lead to poor performance.

[0241] In the DPDK-based deployment of the virtual router 506 ( Figure 5 (Not shown in the image), the virtual router 506 is installed as a user-space application 545 linked to the DPDK library. Specifically, this can result in faster performance than a kernel-based deployment when higher packet rates are present. The physical interface 532 is used by the DPDK's polling-mode driver (PMD), rather than the kernel's interrupt-based driver. The registers of the physical interface 532 can be exposed to user space 545 to make it accessible to the PMD; the physical interface 532 bound in this way is no longer managed by or visible to the host operating system, and the DPDK-based virtual router 506 manages the physical interface 532. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the DPDK data plane of the virtual router 506. Compared to interrupt mode, the nature of this "polling mode" makes packet processing / forwarding by the DPDK data plane of the virtual router 506 more efficient at higher packet rates. Compared to the kernel-mode virtual router 506, there are relatively fewer interrupts and context switches during packet I / O, and in some cases, interrupts and context switches can be avoided together during packet I / O.

[0242] Typically, each container pool in container pools 502A-502B can be assigned one or more virtual network addresses for use within a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by virtual router 506. For example, container pool 502B can be assigned its own third-layer (L3) IP address to send and receive communications; however, container pool 502B may not know the IP address on which container pool 502B of computing device 500 is running. Therefore, the virtual network address may differ from the logical address of the underlying physical computer system (e.g., computing device 500).

[0243] Computing device 500 includes a virtual router agent 514 that controls the coverage of virtual networks within computing device 500 and coordinates the routing of data packets within computing device 500. Typically, virtual router agent 514 communicates with network controller 24, which generates commands to create virtual networks and configure network virtualization endpoints, such as computing device 500, and more specifically, virtual router 506 and virtual network interface 212. By configuring virtual router 506 based on information received from network controller 24, virtual router agent 514 can support configuring network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service link capabilities for orchestration.

[0244] In one example, network packets, such as Layer 3 (L3) IP packets or Layer 2 (L2) Ethernet packets generated and used by containers 529A-529B located within a virtual network domain, can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted over the physical network. The packet transmitted within the virtual network may be referred to herein as an "internal packet," while the physical network packet may be referred to herein as an "external packet" or "tunnel packet." The encapsulation and / or decapsulation of virtual network packets within physical network packets can be performed by the virtual router 506. This function is referred to herein as tunneling and can be used to create one or more overlay networks. Other exemplary tunneling protocols that can be used besides IPinIP include GRE-based Multiprotocol Label Switching (MPLS), UDP-based MPLS, etc. Virtual router 506 performs tunnel encapsulation / decapsulation on packets originating from / delivered to container pool 502, and virtual router 506 exchanges packets with container pool 502 via bridges of bus 542 and / or NIC 530.

[0245] As described above, network controller 24 can provide a logically centralized controller to facilitate the operation of one or more virtual networks. For example, network controller 24 can maintain a routing information base, such as one or more routing tables storing routing information about the physical network and one or more overlay networks. Virtual router 506 implements one or more Virtual Routing and Forwarding Instances (VRFs), such as VRF 222A, for the corresponding virtual network as the corresponding tunnel endpoint. Typically, each VRF stores forwarding information about the corresponding virtual network and identifies where data packets are forwarded and whether the packets are encapsulated in a tunneling protocol, such as using a tunnel header that may include one or more headers from different layers of the virtual network protocol stack. Each VRF may include a network forwarding table storing routing and forwarding information about the virtual network.

[0246] NIC 530 can receive tunnel packets. Virtual router 506 processes the tunnel packets to determine the virtual networks of the source and destination endpoints of the internal packets from the tunnel encapsulation header. Virtual router 506 can strip the Layer 2 header and the tunnel encapsulation header to forward the internal packets only internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN tag or MPLS label, indicating the virtual network (e.g., the virtual network corresponding to VRF 222A). VRF 222A may include forwarding information about the internal packets. For example, VRF 222A may map the destination Layer 3 address of the internal packets to virtual network interface 212. Accordingly, VRF 222A forwards the internal packets via virtual network interface 212 of container pool 502A.

[0247] Container 529A can also initiate internal packets as source virtual network endpoints. For example, container 529A can generate Layer 3 internal packets destined for a destination virtual network endpoint executed by another computing device (i.e., non-computing device 500) or destined for another container. Container 529A can send Layer 3 internal packets to virtual router 506 via a virtual network interface attached to VRF 222A.

[0248] Virtual router 506 receives an internal packet and a Layer 2 header and determines the virtual network of the internal packet. Virtual router 506 can determine the virtual network using any of the virtual network interface implementation technologies described above (e.g., macvlan, veth, etc.). Virtual router 506 uses VRF 222A corresponding to the virtual network of the internal packet to generate an external header for the internal packet, including an external IP header for overlaying the tunnel, and a tunnel encapsulation header identifying the virtual network. Virtual router 506 encapsulates the internal packet using the external header. Virtual router 506 can encapsulate the tunnel packet using a new Layer 2 header with a destination Layer 2 address associated with a device located outside computing device 500 (e.g., TOR switch 16 or a server 12). If located outside computing device 500, virtual router 506 uses physical function 221 to output the tunnel packet with the new Layer 2 header to NIC 530. NIC 530 outputs the packet on its outbound interface. If the destination is another virtual network endpoint executed on computing device 500, virtual router 506 routes the packet to an appropriate virtual network interface of virtual network interfaces 212 and 213.

[0249] In some examples, the controller used for computing device 500 (e.g., Figure 1 The network controller 24) configures the default route in each container pool 502 to cause the container pool to use the virtual router 506 as the initial next hop for outbound packets. In some examples, the NIC 530 is configured with one or more forwarding rules that cause all packets received from the container pool to be switched to the virtual router 506.

[0250] Container pool 502A includes one or more application containers 529A. Container pool 502B includes an instance of Containerized Routing Protocol Daemon (cRPD) 560. Container platform 588 includes container runtime 590, orchestration agent 592, service agent 593, and CNI 570.

[0251] Container engine 590 includes code executed by microprocessor 510. Container runtime 590 can be one or more computer processes. Container engine 590 runs containerized applications in the form of containers 529A-529B. Container engine 590 can represent Dockert, rkt, or other container engines used to manage containers. Typically, container engine 590 receives requests and manages objects such as images, containers, networks, and capacities. An image is a template with instructions for creating containers. A container is an executable instance of an image. Based on instructions from controller agent 592, container engine 590 can obtain an image and instantiate it as an executable container in container pools 502A-502B.

[0252] Service agent 593 includes code executable by microprocessor 510. Service agent 593 can be one or more computer processes. Service agent 593 monitors the addition and removal of services and endpoint objects, and, for example, uses services to maintain the network configuration of computing device 500 to ensure communication between container pools and containers. Service agent 593 can also manage management IP tables to capture traffic to the virtual IP addresses and ports of services and redirect traffic to proxy ports that back up the container pool. Service agent 593 can represent the kube agent of a slave node in a Kubernetes cluster. In some examples, container platform 588 does not include service agent 593, or service agent 593 is disabled by CNI 570 to facilitate the configuration of virtual router 506 and container pool 502.

[0253] The orchestration agent 592 includes code executable by the microprocessor 510. The orchestration agent 592 can be one or more computer processes. The orchestration agent 592 can represent a kubelet on a slave node of a Kubernetes cluster. The orchestration agent 592 is an agent that receives container-specific data from containers and ensures that the container orchestrator is executed by the computing device 500, for example... Figure 1 The orchestrator 23 in the system. Container specification data can be in the form of a manifest file sent from the orchestrator 23 to the orchestration agent 592 or received indirectly via a command-line interface, HTTP endpoint, or HTTP server. Container specification data can be a container pool specification (e.g., PodSpec-YAML (another markup language) or a JSON object describing the container pool) for a container pool 502. Based on the container specification data, the orchestration agent 592 guides the container engine 590 to obtain the container image of container 529 and instantiates the container image for the computing device 500 to execute container 529.

[0254] Orchestration agent 592 instantiates CNI 570 and otherwise invokes CNI 570 to configure one or more virtual network interfaces for each container pool 502. For example, orchestration agent 592 receives container specification data for container pool 502A and bootstraps container engine 590 to create container pool 502A with container 529A based on the container specification data for container pool 502A. Orchestration agent 592 also invokes CNI 570 to configure the virtual network interface for container pool 502A for the virtual network corresponding to VRF 222A. In this example, container pool 502A is a virtual network endpoint of the virtual network corresponding to VRF 222A.

[0255] CNI 570 can obtain interface configuration data for configuring the virtual network interfaces of container pool 502. Virtual router agent 514 operates as a virtual network control plane module that enables network controller 24 to configure virtual router 506. Unlike the orchestration control plane (including container platform 588, e.g., orchestrator 23, for slave and master nodes) which manages provisioning, scheduling, and virtual execution elements, the virtual network control plane (including network controller 24 for slave nodes and virtual router agent 514) manages the configuration of the virtual network partially implemented in the data plane by the virtual router 506 of the slave nodes. Virtual router agent 514 communicates interface configuration data about the virtual network interfaces to CNI 570, enabling orchestration control plane elements (i.e., CNI 570) to configure the virtual network interfaces according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this enables the CNI 570 to obtain interface configuration data for multiple virtual network interfaces used in the container pool and to configure multiple virtual network interfaces, which reduces the communication and resource overhead inherent in calling the individual CNI 570 for configuring each virtual network interface.

[0256] The containerized routing protocol daemon is described in U.S. Patent Application No. 17 / 649,632, filed on February 1, 2022, the entire contents of which are incorporated herein by reference.

[0257] Figure 6 This is a block diagram of an exemplary computing device operating as a compute node in one or more clusters of an SDN architecture system according to the technology disclosed herein. Computing device 1300 may represent one or more physical or virtual servers. In some instances, computing device 1300 may implement one or more master nodes of a given cluster or multiple clusters.

[0258] Although shown and described as being operated by a single computing device 1300, the scheduler 1322, access control manager 305, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration storage 1328 can be distributed among multiple computing devices constituting a computing system or hardware / server cluster. In other words, each of the multiple computing devices can provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, access control manager 305, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, network controller manager 1326, network controller 1324, SDN controller manager 1325, control node 232A, or configuration storage 1328.

[0259] In this example, computing device 1300 includes a bus 1342 coupled to hardware components in the hardware environment of computing device 1300. Bus 1342 is coupled to a network interface card (NIC) 1330, a storage disk 1346, and one or more microprocessors 1310 (hereinafter referred to as "microprocessor 1310"). In some cases, the front-side bus may couple microprocessor 1310 to memory device 1344. In some examples, bus 1342 may couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 may represent a Peripheral Component Interface (PCI) Faster (PCIe) bus 242. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 1342. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

[0260] The microprocessor 1310 may include one or more processors, each of which includes an independent execution unit that executes instructions conforming to an instruction set architecture and stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0261] Disk 1346 represents a computer-readable storage medium, including volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technique for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, disk storage, or other magnetic storage devices, or any other medium capable of storing desired information and accessible to the microprocessor 1310.

[0262] Main memory 1344 includes one or more computer-readable storage media, which may include various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), random access memory (RAM) of flash memory, or any other form of fixed or removable storage medium capable of carrying or storing required program code and program data in the form of instructions or data structures that are accessible to a computer. Main memory 1344 provides a physical address space consisting of addressable memory locations.

[0263] Network interface card (NIC) 1330 includes one or more interfaces 1332 configured to exchange packets using links of the underlying physical network. Interface 1332 may include a port interface card having one or more network ports. NIC 1330 may also include on-card memory, for example, for storing packet data. Direct memory access transfers between NIC 1330 and other devices coupled to bus 1342 can be read from the NIC memory, or direct memory access transfers between NIC 1330 and other devices coupled to bus 1342 can be written to the NIC memory.

[0264] Memory 1344, NIC 1330, storage disk 1346, and microprocessor 1310 provide an operating environment for the software stack of the operating system kernel 1314, which executes in kernel space. For example, kernel 1314 may represent Linux, Berkeley Software Distribution (BSD), another Unix variant's kernel, or a Windows server operating system kernel commercially available from Microsoft. In some instances, the operating system may run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor can encompass a Virtual Machine Manager (VMM). The operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. Kernel 1314 includes a physical drive 1327 using network interface card 1330.

[0265] The computing device 1300 can be coupled to a physical network switching infrastructure, which includes an overlay network of software or virtual routers that extends the switching infrastructure from physical switches to physical servers coupled to the switching infrastructure (such as virtual router 21). The computing device 1300 can use one or more dedicated virtual networks to configure slave nodes of the cluster.

[0266] API server 300A, access control manager 305, scheduler 1322, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, and configuration storage 1328 can implement the master node of the cluster and can be alternatively referred to as the "master control component". The cluster can be a Kubernetes cluster, and the master node can be a Kubernetes master node, in which case the master control component is the Kubernetes master control component.

[0267] Each of the API server 300A, access control manager 305, controller 406A, custom API server 301A, and custom resource controller 302A includes code executable by microprocessor 1310. Custom API server 301A validates and configures data regarding custom resources configured with respect to the SDN architecture. A service can be an abstraction of a logical collection defining container pools and policies for accessing those pools. A set of container pools implementing the service is selected based on the service definition. A service can be partially implemented as a load balancer or otherwise include a load balancer. API server 300A and custom API server 301A can implement a Representational State Transfer (REST) ​​interface to process REST operations and provide a front-end as part of the SDN architecture's configuration plane to the corresponding cluster shared state stored in configuration storage 1328. API server 300A can represent a Kubernetes API server.

[0268] Access control manager 305 is configured to execute the techniques described in this disclosure to generate access control policies for roles. For example, as API server 300A and custom API server 301A receive functions specified by requests and generate access control policies for roles, access control manager 305 can log the execution of functions by API server 300A and custom API server 301A in an audit log. Access control manager 305 can filter and / or parse the audit log to identify multiple resources accessed due to the execution of functions and, for each resource, determine the corresponding one or more types of operations performed on that resource. Therefore, access control manager 305 can create access control policies for roles based at least in part on the parsed audit logs to permit roles to perform corresponding one or more types of operations on each of the multiple resources.

[0269] Configuration storage 1328 is a backup storage for all cluster data. Cluster data may include cluster state and configuration data. Configuration data may also provide a backend for service discovery and / or provide locking services. Configuration storage 1328 may be implemented as a key value store. Configuration storage 1328 may be a central database or a distributed database. Configuration storage 1328 may represent etcd storage. Configuration storage 1328 may represent Kubernetes configuration storage. In some examples, configuration storage 1328 may store access control policies generated using the techniques described in this disclosure and / or other configured access control policies.

[0270] Scheduler 1322 includes code executable by microprocessor 1310. Scheduler 1322 can be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., container pools within containers) and selects slave nodes for running virtual execution elements. Scheduler 1322 can select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, locality, etc. Scheduler 1322 can represent a Kubernetes scheduler.

[0271] Typically, API server 300 can invoke scheduler 1322 to schedule the container pool. Scheduler 1322 can select slave nodes and return an identifier for the selected slave node to API server 300, which can then write the identifier to configuration store 1328 associated with the container pool. API server 300 can invoke orchestration agent for the selected slave node, causing container engine 208 for the selected slave node to obtain the container pool from the storage server and create virtual execution elements on the slave node. The orchestration agent for the selected slave node can update the status of the container pool to API server 300A, which persists this new status to configuration store 1328. In this way, computing device 1300 instantiates a new container pool in computing infrastructure 8.

[0272] Controller manager 1326 includes code executable by microprocessor 1310. Controller manager 1326 can be one or more computer processes. Controller manager 1326 can be embedded in a kernel control loop to monitor the shared state of the cluster by receiving notifications from API server 300A. Controller manager 1326 can attempt to move the cluster state toward a desired state. Exemplary controller 406A and custom resource controller 302A can be managed by controller manager 1326. Other controllers may include replication controllers, endpoint controllers, namespace controllers, and service account controllers. Controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminate container pool garbage collection, cascading deletion garbage collection, node garbage collection, etc. Controller manager 1326 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0273] Network controllers used in the SDN architecture described herein can provide cloud networking for computing architectures operating on top of network infrastructure. Cloud networks can include private clouds for enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) for cloud service providers (CSPs). Private cloud, VPC, and IaaS use cases may involve, for example, references... Figure 1 This describes a multi-tenant virtualized data center. In this scenario, multiple tenants located within the data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is allocated its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual memory; virtual networks). These logical resources are isolated from each other unless explicitly permitted by security policies. The virtual networks located within the data center can also interconnect with physical IP VPNs or L2VPNs.

[0274] A network controller (or "SDN controller") can provide network function virtualization (NFV) for networks such as service edge networks, broadband user management edge networks, and mobile edge networks. NFV involves the orchestration and management of network functions in virtual machines, containers, or other virtual execution elements rather than physical hardware devices, such as firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, and wide area network (WAN) optimization.

[0275] SDN Controller Manager 1325 includes code executable by microprocessor 1310. SDN Controller Manager 1325 can be one or more computer processes. SDN Controller Manager 1325 operates as an interface between orchestration-oriented components (e.g., scheduler 1322, API server 300A and custom API server 301A, controller manager 1326, and configuration store 1328). Typically, SDN Controller Manager 1325 monitors clusters of new Kubernetes native objects (e.g., container pools and services). SDN Controller Manager 1325 can isolate container pools in virtual networks and connect container pools to services.

[0276] The SDN Controller Manager 1325 can run as a container on the master node of a cluster. As described here, in some cases, using the SDN Controller Manager 1325 can disable service proxies on slave nodes (e.g., Kubernetes kube-proxy) to enable full container pool connectivity using virtual routers.

[0277] The components of network controller 24 can operate as a Kubernetes CNI and can support multiple deployment modes. CNI 17 and CNI 750 are compute node interfaces of the entire CNI framework used to manage the Kubernetes network. The deployment modes can be divided into two categories: (1) as an SDN architecture cluster of CNI integrated into the workload Kubernetes cluster; and (2) as an SDN architecture cluster of CNI separated from the workload Kubernetes cluster.

[0278] Integration with workload Kubernetes clusters

[0279] Components of network controller 24 (e.g., custom API server 301, custom resource controller 302, SDN controller manager 1325, and control node 232) run in a management Kubernetes cluster on a master node close to the Kubernetes controller components. In this mode, components of network controller 24 are actually part of the same Kubernetes cluster as the workload.

[0280] Separate from workload Kubernetes cluster

[0281] The components of network controller 24 are executed by a separate library, Bonettis cluster, from a workload Kubernetes cluster.

[0282] The SDN Controller Manager 1325 can use the controller framework for the orchestration platform to listen for (or otherwise monitor) changes to objects defined in the Kubernetes native API and add annotations to some of these objects. Annotations can be labels indicating object characteristics (e.g., "virtual network green") or other identifiers. The SDN Controller Manager 1325 is a component of the SDN architecture; that is, it listens for events from Kubernetes kernel resources (such as container pools, network policies, services, etc.) and translates these events into custom resources configured for the SDN architecture as needed. CNI plugins (e.g., CNI 17, 570) are SDN architecture components that support the Kubernetes network plugin standard: Container Network Interface.

[0283] The SDN Controller Manager 1325 can use the REST interface exposed by the Aggregate API 402 to create network solutions for applications, defining network objects such as virtual networks, virtual network interfaces, and access control policies. For example, the Network Controller 24 component can implement a network solution in a computing infrastructure by configuring one or more virtual networks and virtual network interfaces in a virtual router. (This is just one example of SDN configuration).

[0284] The following exemplary deployment configuration for this application consists of a container pool and the virtual network information of that container pool:

[0285]

[0286] This metadata information can be copied to each container pool replica created by controller manager 1326. When these container pools are notified to SDN controller manager 1325, SDN controller manager 1325 can create the virtual networks listed in the comments ("red network", "blue network", and "default / extns network" in the above embodiment) and create a virtual network interface for each container pool replica (e.g., container pool 202A) for each virtual network using a unique private virtual network address from the cluster-wide address block (e.g., 10.0 / 16) of the virtual network.

[0287] Additional techniques according to this disclosure are described below. Contrail is an exemplary network controller architecture. The Contrail CNI may be a CNI developed for Contrail. The cloud-native Contrail controller may be an example of the network controller described in this disclosure, such as network controller 24.

[0288] Figure 7A This is a block diagram illustrating the control / routing plane configuration of the underlying network and overlay network using the SDN architecture according to the technology disclosed herein. Figure 7B This is a block diagram illustrating a configured virtual network that uses a tunnel connection to a container pool in the underlying network according to the technology disclosed herein.

[0289] The network controller 24 used in the SDN architecture can use a distributed or centralized routing plane architecture. The SDN architecture can use containerized routing protocol daemons (processes).

[0290] In terms of network signaling, the routing plane can operate according to a distributed model, where cRPD runs on every compute node in the cluster. Essentially, this means building intelligence into the compute nodes and involving complex configurations at each node. In this model, route reflectors (RRs) may not make intelligent routing decisions, but are used as relays reflecting routes between nodes. The Distributed Container Routing Protocol Daemon (cRPD) is a usable routing protocol process where each compute node runs its own instance of the routing daemon. Simultaneously, a centralized cRPD master instance can be used as an RR to relay routing information between compute nodes. Routing and configuration intelligence are distributed across nodes using RRs located at a central position.

[0291] Alternatively, the routing plane can operate under a more centralized model, where components of the network controller operate centrally and absorb the intelligence needed to process configuration information, build network topology, and program the forwarding plane into virtual routers. Virtual router agents are local agents that process the information programmed by the network controller. This design leads to a more limited intelligence required at the compute nodes and tends to produce simpler configuration states.

[0292] The centralized control plane provides the following:

[0293] • Allows for a simpler and lighter proxy routing framework. It hides the complexity and limitations of BGP from the proxy. The proxy does not need to understand concepts such as route discriminators and route targets. The proxy simply exchanges prefixes and constructs its corresponding forwarding information.

[0294] There are indeed more control nodes than routes. Control nodes are built on the concept of virtual networks and can generate new routes by using route replication and reorganization (e.g., supporting service links and internal-VM routing features for other use cases).

[0295] • Construct a BUM tree for optimal broadcast and multicast forwarding.

[0296] It should be noted that the control plane has a specific distributed nature. As a control plane that supports distributed functionality, it allows each local virtual router agent to advertise its local routes and subscribe to configurations as needed.

[0297] Therefore, it makes sense to consider control plane design from the perspective of the tool POV and to appropriately use the most suitable tools at hand. Consider the advantages and disadvantages of the combination of Contrail-BGP and cRPD.

[0298] The following functions can be provided by the cRPD or control node of network controller 24.

[0299] Router daemon / process

[0300] Control nodes and cRPDs can be used as routing daemons that implement different protocols and have the ability to program routing information in the forwarding plane.

[0301] cRPD implements routing protocols with a rich routing stack, including Interior Gateway Protocols (IGPs) (e.g., Intermediate System to Intermediate System (IS-IS)), BGP-LU, BGP-CT, SR-MPLS / SRv6, Bidirectional Forwarding Detection (BFD), Path Computation Element Protocol (PCEP), and more. It can also be deployed to provide services such as route reflectors solely for the control plane, and is popular in Internet routing use cases due to these capabilities.

[0302] Control node 232 also implements routing protocols, but primarily based on BGP. Control node 232 understands the overlay network. Control node 232 provides a rich feature set for overlay virtualization and caters to SDN use cases. Overlay features such as virtualization (an abstraction using virtual networks) and service chaining are very popular among telecom and cloud providers. In some cases, cRPD may not include support for this overlay functionality. However, cRPD's rich feature set provides robust support for the underlying network.

[0303] Network Orchestration / Automation

[0304] Routing functionality is only one part of control node 232. Orchestration is the overall component of the overlay network. In addition to providing overlay routing, control node 232 helps model orchestration capabilities and provides network automation. At the heart of control node 232's orchestration capabilities is its ability to model network virtualization using abstractions based on virtual networks (and related objects). Control node 232 interfaces with configuration node 230 to relay configuration information to the control and data planes. Control node 232 also helps build overlay trees for multicast Layer 2 and Layer 3. For example, the control node can build the virtual topology of its clusters used to achieve this. Typically, cRPD does not include this orchestration capability.

[0305] High availability and horizontal scalability

[0306] The control node design is more centralized, while cRPD is more distributed. cRPD worker nodes exist running on each compute node. On the other hand, the control node 232 does not run on compute and can even run on remote clusters (i.e., independently and, in some cases, geographically far from the workload cluster). The control node 232 also provides horizontal scalability for HA and operates in an active-active mode. Compute load is shared among the control nodes 232. cRPD, on the other hand, typically does not provide horizontal scalability. Both the control node 232 and cRPD can provide smooth restarts for HA and can allow headless data plane operation—where virtual routers can continue operating even if the control plane restarts.

[0307] The control plane should be more than just a routing daemon. It should support overlay routing and network orchestration / automation. While cRPD performs as well as routing protocols in managing underlying routes, it typically lacks network orchestration capabilities and doesn't offer robust support for overlay routing.

[0308] Accordingly, in some examples, the SDN architecture can be implemented in, for example... Figures 7A to 7B The compute node shown has cRPD. Figure 7A SDN architecture 700 is shown, which can represent exemplary implementations of SDN architecture 200 or 400. In SDN architecture 700, cRPD 324 runs on compute nodes and provides underlying routing to the forwarding plane while running a centralized (and horizontally scalable) set of control nodes 232 that provide orchestration and coverage services. In some examples, instead of running cRPD 324 on compute nodes, a default gateway can be used.

[0309] The cRPD 324 on the compute node provides rich low-level routing to the forwarding plane by interacting with the virtual router agent 514 through interface 540 (which may be a gRPC interface). The virtual router agent interface can be programmed to route, configured with virtual network interfaces for coverage, and otherwise configured with the virtual router 506. This is described in further detail in U.S. Patent Application No. 17 / 649,632. Meanwhile, one or more control nodes 232 operate as a separate container pool providing coverage services. Thus, the SDN architecture 700 can obtain rich coverage and orchestration provided by control nodes 232 and supplementary modern low-level routing provided by cRPD 324 on the compute node. A separate cRPD controller 720 can be used to configure cRPD 324. cRPD controller 720 can be a device / component management system, network management system, orchestrator, user interface / CLI, or other controller. cRPD 324 runs routing protocols and exchanges routing protocol messages with routers, including other cRPD 324s. Each cRPD 324 can be a centralized routing protocol process and actually operates as a pure software version of the router control plane.

[0310] The enhanced underlying routing provided by cRPD 324 can replace the default gateway in the forwarding plane and provide a rich routing stack for the use cases it can support. In some examples where cRPD 324 is not used, virtual router 506 relies on the default gateway of the underlying routing. In some examples, cRPD 324, as the underlying routing process, is restricted to programming the default inet(6).0 structure using only control plane routing information. In this example, the non-default overlay VRF can be programmed by control node 232.

[0311] Figures 7A to 7B The dual routing / control plane solution described above is shown. Figure 7A In one respect, similar to how the router control plane programs the router forwarding / data plane, the cRPD 324 provides underlying routing / forwarding information to the virtual router agent 514.

[0312] like Figure 7B As shown, cRPD 324 switching can be used to create routing information for tunneling through the underlying network 702 of the VRF. Tunnel 710 is an example and connects server 12A to server 12X via virtual router 506. Tunnel 710 can represent a segmented routing (SR) or SRv6 tunnel, a generic route encapsulation (GRE) tunnel, an IP-in-IP tunnel, an LSP, or other tunnels. Control node 232 uses tunnel 710 to create virtual network 712, which connects server 12A, which is attached to the VRF, to container pool 22 of server 12X.

[0313] As described above, cRPD 324 and virtual router agent 514 can exchange routing information using the gRPC interface, and virtual router agent 514 can program virtual router 506 through configuration using the gRPC interface. It should also be noted that control node 232 can be used for overlay and orchestration, while cRPD 324 can be used to manage underlying routing protocols. Virtual router agent 514 can use the gRPC interface with cRPD 324 while communicating with the control node and Domain Name Service (DNS) using XMPP.

[0314] Because workers may be running on various compute nodes, the gRPC model performs well for cRPD 324. Furthermore, the virtual router proxy 514 acts as a gRPC server exposing clients (cRPD 324) for programming routing and configuration information (at the lower level). Therefore, gRPC is an attractive solution when compared to XMPP. Specifically, gRPC transmits data as a binary stream and incurs no overhead when encoding / decoding data sent via gRPC.

[0315] In some examples, control node 232 can interface with virtual router agent 514 using XMPP. Since virtual router agent 514 acts as a gRPC server and cRPD 324 acts as a gRPC client, this means the client (cRPD) needs to initiate a connection to the server (vRouter agent). In the SDN architecture 700, virtual router agent 514 selects the set of control nodes 232 it subscribes to (because multiple control nodes exist). In this respect, control node 232 acts as a server and virtual router agent 514 connects and subscribes for updates as a client.

[0316] Control node 232 needs to select the virtual router agent 514 it needs to connect to via gRPC and then subscribe as a client. Since control node 232 does not run on every compute node, this requires implementing an algorithm to select the virtual router agent 514 it can subscribe to. Furthermore, control nodes 232 need to synchronize this information with each other. This further complicates matters when a restart occurs, requiring synchronization between control nodes 232 to select the agent they serve. Features such as smooth restart (GP) and fast convergence have already been implemented on top of XMPP. XMPP is already lightweight and efficient. Therefore, for communication between control node 232 and virtual router agent 514, XMPP may be superior to gRPC.

[0317] The additional enhancements used by control node 232 are as follows. HA and horizontal scalability have three control nodes. As with any routing platform, two control nodes 232 are sufficient to meet HA requirements. This is advantageous in many cases. (However, one or more control nodes 232 can be used). For example, it provides a more deterministic infrastructure and conforms to best practices for standard routing. Each virtual router agent 514 is attached to a unique pair of control nodes 232 to avoid randomness. Debugging may be simpler with two control nodes 232. Furthermore, using only two control nodes 232 simplifies edge replication for building multicast / broadcast trees. Currently, because virtual router agents 514 are only connected to two of the three control nodes, all control nodes may not have a complete picture of the tree at any given time and rely on BGP to synchronize the state between control nodes. The presence of three control nodes 232 exacerbates this situation because virtual router agents 514 can randomly select two. If only two control nodes 232 exist, each virtual router agent 514 would connect to the same control nodes. This further indicates that control node 232 does not need to rely on BGP to synchronize its state and has the same picture of the multicast tree.

[0318] SDN Architecture 200 can provide ingress replication as an alternative to edge replication and offers users an option. Ingress replication can be viewed as a special degenerate case of a general overlay multicast tree. However, in practice, the signaling of an ingress-replicated tree is simpler than that of a general overlay multicast tree. With ingress replication, each virtual router 21 terminates in a tree with itself as the root and each other vrouter as the leaf. A failed virtual router 21 should theoretically not require tree rebuilding. It should be noted that the performance of ingress replication degrades with larger clusters. However, ingress replication performs well for smaller clusters. Furthermore, multicast is not a popular and universal requirement for many customers. It is primarily limited to transmitting broadcast BUM traffic that occurs only initially.

[0319] Enhanced configuration processing module

[0320] In a typical SDN architecture, the network controller handles the orchestration of all use cases. Configuration nodes translate intents into configuration objects based on a data model and write these objects to a database (e.g., Cassandra). In some cases, such as simultaneously, notifications are sent to all clients awaiting configuration via RabbitMQ.

[0321] The control node not only functions as a BGP speaker but also has a configuration processing module that reads configuration objects from the database in the following ways: First, when the control node starts (or restarts), it connects to the database and reads all configurations directly from it. Second, the control node can also be a message client. When there is an update to a configuration object, the control node receives a message notification listing the updated objects. This again causes the configuration processing module to read the objects from the database.

[0322] The configuration processing module reads configuration objects from the control plane (BGP-related configuration) and the vRouter forwarding plane. The configuration can be stored as a graph with objects as nodes and relationships as links. This graph can then be downloaded to the client (BGP / cRPD and / or vRouter agent).

[0323] According to the techniques disclosed herein, in some examples, the conventionally configured API server and messaging service are replaced by kubeapi-servers (API server 300 and custom API server 301), and the previous Cassandra database is replaced by etcd in Kubernetes. Due to this change, clients interested in configuration objects can directly specify monitoring on the etcd database to receive updates, rather than relying on RabbitMQ notifications.

[0324] CRPD controller orchestration

[0325] BGP configuration can be provided to cRPD 324. In some examples, cRPD controller 720 can be a Kubernetes controller that caters to the development of its own controller and implements the CRDs required for orchestration and provisioning of cRPD 324, with its own controller catering to the Kubernetes space.

[0326] Distributed configuration processing

[0327] As mentioned earlier in this section, the configuration processing module can be part of the control node 232. The configuration processing module reads the configuration directly from the database, converts the data into JSON format, and stores it as a graph in its local IFMAP database, with objects as nodes and relationships between them as links. This graph is then downloaded via XMPP to the virtual router agent 514 of interest on the compute node. The virtual router agent 514 also locally constructs an IFMAP-based dependency graph to store these objects.

[0328] By enabling the virtual router agent 514 to monitor directly on the etcd server in API server 300, the need for IFMAP as an intermediate module and the storage dependency graph can be avoided. The same model can be used for cRPD 324 running on the compute node. This avoids the need for an IFMAP-XMPP configuration channel. The Kubernetes configuration client (for control node 232) can be used as part of this configuration. The virtual router agent can also use this client.

[0329] However, this can increase the number of clients reading configurations from the etcd server, especially in clusters with hundreds of compute nodes. Adding more monitors eventually leads to a decrease in write rates and event rates that fall below ideal levels. etcd's gRPC agent replays events from one server monitor to multiple client monitors. The gRPC agent merges multiple client monitors (c-monitors) with the same key or range into a single monitor (s-monitor) connected to the etcd server. The agent broadcasts all events from the s-monitor to its c-monitor. Assuming N clients monitor the same key, a single gRPC agent can reduce the monitoring load on the etcd server from N to 1. Users can deploy multiple gRPC agents to further distribute the server load. These clients share a single server monitor; the agent effectively offloads resource pressure from the core cluster. By adding agents, etcd can serve one million events per second.

[0330] DNS / Naming in SDN Architecture

[0331] In the previous architecture, DNS services were provided by the jointly working contrail-dns and contrail-naming processes to provide DNS services to VMs in the network. Naming acted as the DNS server providing an implementation of the BIND protocol, and contrail-dns received updates from the vrouter-proxy and pushed these records to Naming.

[0332] The system supports four DNS modes, and IPAM configuration allows you to select the desired DNS mode.

[0333] 1. None - No DNS support for VMs.

[0334] 2. Default DNS Server - DNS resolution for the VM is performed based on the name server configuration in the server infrastructure. When the VM receives a DHCP response, the subnet default gateway is configured as the VM's DNS server. DNS requests sent by the VM to this default gateway are resolved via the (structure) name server configured on the corresponding compute node, and the response is sent back to the VM.

[0335] 3. Tenant DNS Server - Tenants can utilize this mode to use their own DNS server. A list of servers can be configured in IPAM, and then the server can be sent as a DNS server to the VM in a DHCP response. DNS requests sent by the VM are routed as any other data packet based on available routing information.

[0336] 4. Virtual DNS Server - In this mode, the system supports virtual DNS servers, thus providing DNS servers to resolve DNS requests from VMs. Those skilled in the art can define multiple virtual DNS servers under each domain in the system. Each virtual DNS server is an authoritative server for the configured DNS domain.

[0337] The SDN architecture described here is effective for the DNS services it provides. Customers in the cloud-native world benefit from a variety of DNS services. However, with the migration to next-generation Kubernetes-based architectures, the SDN architecture can replace the core DNS using any DNS service.

[0338] Data plane

[0339] The data plane consists of two components: Virtual Router Agent 514 (aka Agent) and Virtual Router Forwarding Plane 506 (also known as DPDK vRouter / Kernel vRouter). Agent 514 in the SDN architecture solution manages the data plane components. Agent 514 establishes XMPP neighbor relationships with the two control nodes 232, and then exchanges routing information with them. Agent 514 also dynamically generates flow entries and injects them into Virtual Router 506. This provides Virtual Router 506 with instructions on how to forward packets.

[0340] The responsibilities of agent 514 may include: interfacing with control node 232 to obtain configuration; translating the received configuration into a form that the data path can understand (e.g., translating a data model from IFMAP into a data model used by the data path); interfacing with control node 232 to manage routes; and collecting statistics from the data path and outputting the statistics to the monitoring solution.

[0341] The Virtual Router 506 implements data plane functionality that allows virtual network interfaces to be associated with VRFs. Each VRF has its own forwarding and flow tables, while the MPLS and VXLAN tables are global within the Virtual Router 506. The forwarding table can contain routes for the destination's IP and MAC addresses, and IP-to-MAC associations are used to provide proxy ARP capabilities. When a VM / container interface is started and is only locally important to the vRouter, the label value from the MPLS table is selected by the Virtual Router 506. The VXLAN network identifier is global for all VRFs of the same virtual network across different Virtual Routers 506 within a cross-domain.

[0342] In some examples, each virtual network has a default gateway address assigned to it, and each VM or container interface receives this address in the DHCP response it receives during initialization. When a workload sends packets to an address outside its subnet, an ARP for the MAC address is mapped to the gateway's IP address, and the virtual router 506 responds with its own MAC address. Thus, the virtual router 506 can support a fully distributed default gateway function for all virtual networks.

[0343] Below is an example of packet flow forwarding implemented by Virtual Router 506.

[0344] Packet flow between VM / container interfaces in the same subnet.

[0345] Worker nodes can be VMs or container interfaces. In some examples, packet processing is performed as follows:

[0346] The VMI / container interface needs to send packets to VM2, so the virtual router 506 first queries its own DNS cache to obtain the IP address. However, since this is the first packet, there is no entry.

[0347] • When its interface starts up, VM1 sends a DNS request to the DNS server address provided in the DHCP response.

[0348] The virtual router 506 intercepts DNS requests and forwards them to the DNS server running in the SDN architecture controller.

[0349] The DNS server in the controller responds with the IP address of VM2.

[0350] Virtual router 506 sends a DNS response to VM1.

[0351] VM1 needs to form an Ethernet frame, therefore, it needs VM2's MAC address. VM1 checks its own ARP cache, but since this is the first packet, there is no entry.

[0352] VM1 sends an ARP request.

[0353] The virtual router 506 intercepts the ARP request and looks up the MAC address of IP-VM2 in its own forwarding table, and finds the association in the L2 / L3 routes sent by the controller for the MAC address of VM2.

[0354] Virtual router 506 uses VM2's MAC address to send an ARP reply to VM1.

[0355] • A TCP timeout occurred in the network stack of VM1.

[0356] The network stack of VM1 re-attempts to send packets, and at this time, it finds the MAC address of VM2 in the ARP cache, is able to form an Ethernet frame, and sends the Ethernet frame out.

[0357] Virtual router 506 queries the MAC address of VM2 and finds the encapsulation route. Virtual router 506 constructs the external header and sends the generated packet to server S2.

[0358] The virtual router 506, located on server S2, decapsulates the packets and looks up MPLS tags to identify the virtual interface, then sends the initial Ethernet frame to that virtual interface. The Ethernet frame is sent to the interface and received by VM2.

[0359] Packet streams between VMs located in different subnets

[0360] In some examples, the order in which packets are sent to destinations in different subnets is the same, except that virtual router 506 responds as the default gateway. VM1 sends packets with the MAC address of the default gateway in Ethernet frames, which is provided in the DHCP response supplied by virtual router 506 when VM1 starts up. When VM1 makes an ARP request for the gateway IP address, virtual router 506 responds using its own MAC address. When VM1 sends Ethernet frames using the gateway MAC address, virtual router 506 uses the destination IP address of the packets within the frame to look up the forwarding table in the VRF to find a route to the host running at the destination via the encapsulated tunnel.

[0361] Figure 10 This is a virtualized block diagram illustrating exemplary relationships between access control policies, roles, and subjects. For example... Figure 10 As shown, the visualization 1400 can provide a diagram of access control policies associated with roles and role binding, which binds subjects to access control policies associated with roles.

[0362] The namespace-scoped access control policy 1420 and the namespace-scoring access control policy 1422 can be namespace-scoped, meaning they can be isolated to allow access to resources within namespace 1402 within the cluster. Meanwhile, the cluster-scoped access control policy 1424 can access resources within the cluster, including resources outside namespace 1402.

[0363] Role 1414 is permitted to access resources within namespace 1402 according to namespace-wide access control policy 1420, and ClusterRole 1416 is permitted to access resources within namespace 1402 according to namespace-wide access control policy 1422. Meanwhile, ClusterRole 1418 is permitted to access resources within the cluster according to cluster-wide access control policy 1424.

[0364] Role binding grants permissions (e.g., role access control policies) to a user or a group of users that are defined by a role. Therefore, RoleBinding 1408 can bind subject 1406 to namespace-wide access control policy 1420 for role 1414. RoleBinding 1410 can bind subject 1406 to namespace-wide access control policy 1422 for ClusterRole 1416, and ClusterRoleBinding 1412 can bind subject 1406 to cluster-wide access control policy 1424 for ClusterRole 1418. If a role binding such as RoleBinding 1408 is bound to a missing subject, such as if RoleBinding 1408 were not bound to any user, visualization 1400 can provide visualization of the missing subject 1404 bound by RoleBinding 1408. Computing devices, such as one or more configuration nodes 230 of network controller 24, can output the visualization 1400 of access control policies associated with roles and role bindings of access control policies that bind subjects to roles associated with roles, to be displayed at, for example, user interface 244, so that users (e.g., administrators) can visualize the access control policies of SDN architecture 200.

[0365] Figure 11 This is a flowchart illustrating exemplary operation of the network controller 24 in an SDN architecture 200 according to the technology of this disclosure. For convenience, refer to... Figures 1 to 10 right Figure 11 Describe it.

[0366] exist Figure 11 In the example, one or more configuration nodes 230 of network controller 24 can receive requests to generate access control policies (1500) regarding roles in the container orchestration system. These configuration nodes 230 include an application programming interface (API) server 300 to handle requests for operations on local resources related to the container orchestration system, and a custom API server 301 to handle requests for operations on custom resources related to the SDN architecture configuration. Requests may specify multiple functionalities of the aggregated API 402 provided by the custom API server 301 and the API server 300.

[0367] One or more configuration nodes 230 of network controller 24 may perform multiple functions (1502). One or more configuration nodes 230 of network controller 24 may log the execution of these multiple functions in an audit log (1504). One or more configuration nodes 230 of network controller 24 may parse the audit log to identify multiple resources of the container orchestration system accessed due to the execution of these multiple functions, and for each of these multiple resources, determine one or more types of operations corresponding to the multiple actions performed on the respective resource due to the execution of these multiple functions (1506). Network controller 24 may create access control policies for roles, at least in part, based on the parsed audit logs, to permit roles to perform corresponding one or more types of operations on each of these multiple resources (1508).

[0368] The aspects of this disclosure include the following examples.

[0369] Example 1: A network controller for a software-defined networking (SDN) architecture system, the network controller comprising: a processing circuit; and one or more configuration nodes configured to be executed by the processing circuit, wherein the one or more configuration nodes include an application programming interface (API) server for processing requests for operations on local resources of a container orchestration system, and the one or more configuration nodes include a custom API server for processing requests for operations on custom resources configured in the SDN architecture; to: receive a request; generate an access control policy for a role in the container orchestration system, wherein the request specifies multiple functions provided by the custom API server and an aggregate API provided by the API server; execute the multiple functions; log the execution of the multiple functions in an audit log; parse the audit log to determine multiple resources of the container orchestration system accessed due to the execution of the multiple functions and, for each of the multiple resources, determine one or more types of operations corresponding to multiple actions performed on the corresponding resource due to the execution of the multiple functions; and create an access control policy for a role, at least in part based on the parsed audit log, to permit the role to perform one or more types of operations on each of the multiple resources.

[0370] Example 2: According to the network controller of Example 1, wherein, in order to perform the plurality of functions, one or more configuration nodes are further configured to be executed by processing circuitry to: perform a first corresponding one or more operations indicated by the plurality of functions for each of the first one or more resources indicated by the plurality of functions; determine a second corresponding one or more operations to be performed for each of the second one or more resources not indicated by the plurality of functions based at least in part on the first corresponding one or more operations performed for each of the first one or more resources; and perform the second corresponding one or more operations for each of the second one or more resources.

[0371] Example 3: A network controller according to any one of Examples 1 and 2, wherein, in order to record the execution of the plurality of functions, one or more configuration nodes are further configured to be executed by processing circuitry to: record, for each of the plurality of functions, an event indicating one or more resources accessed by executing the function and an event indicating a corresponding one or more types of operation performed on each of the one or more resources by executing the function.

[0372] Example 4: A network controller according to any one of Examples 1 to 3, wherein, in order to parse the audit logs, one or more configuration nodes are further configured by processing circuitry to: filter the audit logs at least in part based on the timestamps of events recorded in the audit logs.

[0373] Example 5: A network controller according to any one of Examples 1 to 4, wherein, in order to parse the audit logs, one or more configuration nodes are further configured by processing circuitry to: filter the audit logs at least in part based on the namespace of the events recorded in the audit logs.

[0374] Example 6: A network controller according to any one of Examples 1 to 5, wherein, in order to parse the audit logs, one or more configuration nodes are further configured to be executed by the processing circuitry to:

[0375] The association between each of the multiple resources and one or more corresponding types of operations is determined based on the audit logs.

[0376] Example 7: A network controller according to any one of Examples 1 to 6, wherein one or more configuration nodes are further configured to be executed by processing circuitry to: validate a role's access control policy at least in part based on a comparison of the role's access control policy with the role's configured access control policy.

[0377] Example 8: A network controller according to any one of Examples 1 to 7, wherein the functionality of the aggregation API includes requests for operations on one or more instances of custom resources of an SDN architecture configuration, wherein each custom resource of the SDN architecture configuration corresponds to a type of configuration object in the SDN architecture system.

[0378] Example 9: A network controller according to any one of Examples 1 to 8, wherein the functionality of the aggregation API includes requests for operations on instances of one or more local resources of the container orchestration system.

[0379] Example 10: A network controller according to any one of Examples 1 to 9, wherein the container orchestration system includes Kubernetes.

[0380] Example 11: A network controller according to any one of Examples 1 to 10, wherein each of the corresponding one or more types of operations includes one or more of the CRUP operations of creating, reading, updating, and deleting.

[0381] Example 12: A method comprising: receiving a request by processing circuitry of a network controller for a software-defined networking (SDN) architecture system to generate an access control policy for a role in a container orchestration system, wherein the network controller includes an application programming interface (API) server for processing requests for operations on local resources of the container orchestration system, and the network controller includes a custom API server for processing requests for operations on custom resources configured in the SDN architecture, and wherein the request specifies multiple functions provided by the custom API server and an aggregated API provided by the API server; executing the multiple functions by the processing circuitry; logging the execution of the multiple functions in an audit log by the processing circuitry; parsing the audit log to determine multiple resources of the container orchestration system accessed due to the execution of the multiple functions and determining, for each of the multiple resources, a corresponding one or more types of operation for multiple actions performed on the corresponding resource due to the execution of the multiple functions; and creating an access control policy for a role by the processing circuitry and at least in part based on the parsed audit log to permit the role to perform the corresponding one or more types of operation on each of the multiple resources.

[0382] Example 13: According to the method of Example 12, wherein performing the plurality of functions further comprises: performing a first corresponding one or more operations indicated by the plurality of functions on each of the first one or more resources indicated by the plurality of functions by the processing circuit; determining a second corresponding one or more operations to be performed on each of the second one or more resources not indicated by the plurality of functions by the processing circuit and at least in part based on the first corresponding one or more operations performed on each of the first one or more resources; and performing the second corresponding one or more operations on each of the second one or more resources by the processing circuit.

[0383] Example 14: The method according to any one of Examples 12 and 13, wherein recording the execution of the plurality of functions further comprises: by the processing circuitry and for each of the plurality of functions, recording an event indicating one or more resources accessed by executing the function and indicating a corresponding one or more types of operation performed on each of the one or more resources by executing the function.

[0384] Example 15: The method according to any one of Examples 12 to 14, wherein parsing the audit log further includes: filtering the audit log by a processing circuit based at least in part on the timestamps of events recorded in the audit log.

[0385] Example 16: The method according to any one of Examples 12 to 15, wherein parsing the audit log further includes: filtering the audit log by processing circuitry based at least in part on the namespace of the events recorded in the audit log.

[0386] Example 17: The method according to any one of Examples 12 to 16, wherein parsing the audit log further includes: determining, by processing circuitry and based on the audit log, the association between each of the multiple resources and a corresponding one or more types of operation.

[0387] Example 18: The method according to any one of Examples 12 to 17 further includes: verifying the access control policy of a role by a comparison of the access control policy of the role at least in part based on the processing circuitry with the configured access control policy of the role.

[0388] Example 19: According to any one of Examples 12 to 18, the functionality of the aggregation API includes requests for operations on one or more instances of custom resources of an SDN architecture configuration, wherein each custom resource of the SDN architecture configuration corresponds to a type of configuration object in the SDN architecture system.

[0389] Example 20: A non-transitory computer-readable medium comprising: receiving a request; generating an access control policy for a role in a container orchestration system, wherein the request specifies multiple functions provided by a custom API server and an aggregate API provided by the API server; executing the multiple functions; logging the execution of the multiple functions in an audit log; parsing the audit log to determine multiple resources of the container orchestration system accessed as a result of executing the multiple functions and, for each of the multiple resources, determining a corresponding one or more types of operation for a plurality of actions performed on the respective resource as a result of executing the multiple functions; and creating an access control policy for a role, at least in part based on the parsed audit log, to permit the role to perform a corresponding one or more types of operation on each of the multiple resources.

[0390] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. The individual features described as modules, units, or components can be implemented together in an integrated logic device, or as discrete, interoperable logic devices or other hardware devices. In some cases, the individual features of an electronic circuit can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0391] If implemented in hardware, this disclosure may relate to devices such as processors or integrated circuit devices such as integrated circuit chips or chipsets. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store the instructions executed by the processor.

[0392] Computer-readable media can form part of a computer program product that includes packaging material. Computer-readable media can include computer data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0393] In some examples, computer-readable storage media may include non-volatile media. The term "non-volatile" may indicate that the storage media is not included in a carrier wave or propagating signal. In certain examples, non-volatile storage media may store data that changes over time (e.g., stored in RAM or cache).

[0394] The code or instructions can be software and / or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functions described in this disclosure can be set within software or hardware modules.< / none> < / none>

Claims

1. A network controller for a software-defined networking (SDN) architecture system, the network controller comprising: Processing circuitry; as well as One or more configuration nodes are configured to be executed by the processing circuitry, wherein the one or more configuration nodes include an application programming interface (API) server to process requests for operations on local resources of the container orchestration system, and the one or more configuration nodes include a custom API server to process requests for operations on custom resources of the SDN architecture configuration, in order to: Receive a request and generate an access control policy for roles in the container orchestration system, wherein the request specifies multiple functions of the aggregated API provided by the custom API server and the application programming interface API server; Perform the aforementioned multiple functions; The execution of these multiple functions is recorded in the audit log; The audit logs are parsed to identify various resources of the container orchestration system accessed due to the execution of the multiple functions, and for each of the multiple resources, one or more types of operations corresponding to multiple actions performed on the corresponding resource due to the execution of the multiple functions are identified; and The access control policy for the role is created at least in part based on the parsed audit logs to allow the role to perform the corresponding one or more types of operations on each of the multiple resources.

2. The network controller according to claim 1, wherein, In order to perform the multiple functions, the one or more configuration nodes are further configured to be executed by the processing circuitry to: For each of the first one or more resources indicated by the plurality of functions, perform one or more corresponding operations indicated by the plurality of functions; At least in part based on the first corresponding one or more operations performed on each of the first one or more resources, a second corresponding one or more operations performed on each of the first one or more resources that are not indicated by the plurality of functions are determined; and For each of the second or one type of resources, perform the second corresponding one or more operations.

3. The network controller according to claim 1, wherein, In order to record the execution of the multiple functions, the one or more configuration nodes are further configured to be executed by the processing circuitry, in order to: For each of the plurality of functions, an event is recorded that indicates one or more resources accessed by executing the function and that the corresponding one or more types of operation is performed on each of the one or more resources by executing the function.

4. The network controller according to claim 1, wherein, To parse the audit logs, the one or more configuration nodes are further configured to be executed by the processing circuitry to: The audit logs are filtered at least in part based on the timestamps of the events recorded in them.

5. The network controller according to claim 1, wherein, To parse the audit logs, the one or more configuration nodes are further configured to be executed by the processing circuitry to: The audit logs are filtered at least in part based on the namespaces of the events recorded in them.

6. The network controller according to claim 1, wherein, To parse the audit logs, the one or more configuration nodes are further configured to be executed by the processing circuitry to: Based on the audit logs, the association between each of the multiple resources and the corresponding one or more types of operations is determined.

7. The network controller according to any one of claims 1 to 6, wherein, The one or more configuration nodes are further configured to be executed by the processing circuitry to: The access control policy of the role is verified by at least partly comparing the access control policy of the role with the configured access control policy of the role.

8. The network controller according to any one of claims 1 to 6, wherein, The functionality of the aggregated API includes requests for operations on one or more instances of custom resources related to the SDN architecture configuration, wherein each custom resource in the SDN architecture configuration corresponds to a type of configuration object in the SDN architecture system.

9. The network controller according to any one of claims 1 to 6, wherein, The functionality of the aggregate API includes requests for operations on instances of one or more local resources related to the container orchestration system.

10. The network controller according to any one of claims 1 to 6, wherein, The container orchestration system includes Kubernetes.

11. The network controller according to any one of claims 1 to 6, wherein, Each of the corresponding one or more types of operations includes one or more of the CRUD operations: create, read, update, and delete.

12. A method for role-based access control, the method comprising: A request is received by the processing circuitry of a network controller for a software-defined networking (SDN) architecture system to generate an access control policy for roles in a container orchestration system. The network controller includes an application programming interface (API) server to process requests for operations on local resources of the container orchestration system, and a custom API server to process requests for operations on custom resources configured in the SDN architecture. The request specifies multiple functions of an aggregated API provided by the custom API server and the API server. The processing circuit performs the multiple functions; The processing circuit records the execution of the multiple functions in the audit log; The processing circuit parses the audit logs to identify various resources of the container orchestration system accessed due to the execution of the multiple functions, and for each of the multiple resources, determines one or more types of operations corresponding to the multiple actions performed on the corresponding resource due to the execution of the multiple functions; and The access control policy for the role is created by the processing circuitry and at least in part based on the parsed audit logs, to permit the role to perform the corresponding one or more types of operations on each of the multiple resources.

13. The method according to claim 12, wherein, Performing the aforementioned multiple functions further includes: The processing circuit performs one or more corresponding operations indicated by the plurality of functions on each of the first one or more resources indicated by the plurality of functions; The processing circuitry, and at least in part based on the first corresponding one or more operations performed on each of the first one or more resources, determines a second corresponding one or more operations to be performed on each of the second one or more resources that are not indicated by the plurality of functions; and The processing circuit performs one or more corresponding operations on each of the second or more resources.

14. The method according to claim 12, wherein, Recording the execution of the aforementioned multiple functions further includes: The processing circuit records, for each of the plurality of functions, an event indicating one or more resources accessed by executing the function and an event indicating the corresponding one or more types of operation performed on each of the one or more resources by executing the function.

15. The method according to claim 12, wherein, Parsing the audit logs further includes: The processing circuit filters the audit log based at least in part on the timestamps of the events recorded in the audit log.

16. The method according to claim 12, wherein, Parsing the audit logs further includes: The processing circuit filters the audit log based at least in part on the namespace of the events recorded in the audit log.

17. The method according to claim 12, wherein, Parsing the audit logs further includes: The processing circuitry, based on the audit logs, determines the association between each of the various resources and the corresponding one or more types of operations.

18. The method of claim 12, further comprising: The access control policy of the role is verified by the processing circuit based at least in part on a comparison between the access control policy of the role and the configured access control policy of the role.

19. The method according to any one of claims 12 to 18, wherein, The functionality of the aggregated API includes requests for operations on one or more instances of custom resources related to the SDN architecture configuration, wherein each custom resource in the SDN architecture configuration corresponds to a type of configuration object in the SDN architecture system.

20. A non-transitory computer-readable medium comprising instructions for causing processing circuitry of a network controller to execute one or more configuration nodes, the one or more configuration nodes including an application programming interface (API) server to process requests for operations on local resources relating to a container orchestration system, and the one or more configuration nodes including a custom API server to process requests for operations on custom resources relating to SDN architecture configuration, to: Receive a request to generate an access control policy for roles in the container orchestration system, wherein, The request specifies multiple functions of the aggregated API provided by the custom API server and the application programming interface API server; Perform the aforementioned multiple functions; The execution of these multiple functions is recorded in the audit log; The audit logs are parsed to identify various resources of the container orchestration system accessed due to the execution of the multiple functions, and for each of the multiple resources, one or more types of operations corresponding to multiple actions performed on the corresponding resource due to the execution of the multiple functions are identified; and The access control policy for the role is created, at least in part based on the parsed audit logs, to permit the role to perform the corresponding one or more types of operations on each of the multiple resources.

Images