Systems and methods for user customization and automated operations on software-defined networks

Abstract

Embodiments of the present disclosure relate to systems and methods for user customization and automated operations on software-defined networks. Techniques for defining and executing device-independent commands on a network having multiple network devices are described. In some examples, the controller includes a graphical user interface. The controller displays, via the graphical user interface, network devices that support device-independent commands selected from one or more device-independent commands, where each device-independent command performs one or more operations on the supported network devices. The controller receives, via the graphical user interface, user input that selects two or more of the displayed network devices; and performs one or more operations of the selected device-independent command on the selected network devices. Performing includes: performing tasks associated with each network device, where the tasks perform one or more operations on each corresponding network device when executed.

Description

[0001] This application is a divisional application of the invention patent application with the application date of June 27, 2019, application number 201910566508.0, and invention title "System and Method for User Customization and Automated Operations on Software-Defined Networks". Technical Field

[0002] The technology of the present disclosure generally relates to computer networks, and more particularly, to the configuration of virtual networks. Background Art

[0003] Virtualized networks are becoming a core foundation of modern information technology (IT) infrastructure. For example, modern data centers use virtualized environments where virtual computing nodes such as virtual machines or containers (also referred to as virtual execution elements) are deployed and executed on the underlying computing platforms of the physical computing devices of the network.

[0004] Large-scale data centers such as cloud data center environments employ a collection of network servers to provide computing and / or storage capacity to subscribers running various applications. For example, a data center can host infrastructure devices such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switch fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure spread across the globe, where subscriber support devices are located in facilities at various physical hosting sites.

[0005] Virtualization offers several advantages. One advantage is that virtualization can significantly improve efficiency. With the emergence of multi-core microprocessor architectures where each physical CPU has a large number of cores, servers are becoming more and more powerful; power can be easily and efficiently distributed across applications and subscribers in a virtual environment. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become replaceable resources, such as in cloud-based computing environments, the provisioning and management of the computing infrastructure become easier. Therefore, enterprise IT staff generally prefer to use virtualized computing clusters in data centers because, in addition to computing efficiency, they also simplify management and increase the return on investment (ROI).

[0006] Software-defined network (SDN) deployments are increasingly being adopted in virtualized data centers to workloads. These methods simplify the management of virtual and physical computing devices, virtual and physical storage systems, and virtual and physical networks by abstracting the complexity of the underlying infrastructure. A software-defined data center (SDDC) is a data center that uses software-defined networks to virtualize the underlying infrastructure and deliver various aspects of the virtualized infrastructure as services. Summary of the Invention

[0007] In general, software-defined data centers use intent models to automate the day-to-day tasks for delivering multi-tenant services over a data center (DC) fabric. However, intent-based automation tools do not allow system users to implement specific operations in a traditional element management system (EMS). Currently, users and administrators may need to compromise between the ease of control provided by SDN abstractions (intents) and the greater control provided by low-level operations performed via the command line interface (CLI) or traditional EMS.

[0008] This disclosure describes techniques for implementing device-agnostic commands across different types of network devices, which provide an institution for users and administrators to implement specific operations on heterogeneous groups of devices in a network. In various examples, job templates, task collections, and input, output, and input_ui modes can be used to define device-agnostic commands. The devices that support each device-agnostic command are defined by the tasks required to execute the device-agnostic command on the device and the templates defined for the device. The result is an advantageous way to execute device-agnostic commands across network devices from multiple product vendors while achieving similar results. For example, operations on a device family can be expressed as tasks and templates using Ansible scripts and Jinja2 templates, respectively. In an example method, a graphical user interface on a virtual network controller provides an institution for defining common operations and applying device-agnostic commands to devices based on representative tasks and template files. The result provides flexibility for device-agnostic testing across heterogeneous networks.

[0009] In one example, a method includes: displaying, via a graphical user interface, network devices that support a device-agnostic command selected from one or more device-agnostic commands, where each device-agnostic command performs one or more operations on the supported network devices; receiving, via the graphical user interface, user input that selects two or more of the displayed network devices, where each network device has a network device family, where one of the two or more selected network devices is from a first device family and responds to a first set of device commands associated with the first network device family, and where another of the two or more selected network devices is from a second device family and responds to a second different set of device commands associated with the second network device family; and performing one or more operations of the selected device-agnostic command on the selected network devices, where performing the one or more operations includes: performing tasks on each of the selected network devices based on commands from a set of commands associated with the network device family of the corresponding selected network device, where the task, when executed, performs one or more operations on the corresponding selected network device.

[0010] In one example, a virtual network controller includes a memory; a graphical user interface; and one or more processors coupled to the memory and the graphical user interface, wherein the memory includes instructions that, when executed by the one or more processors, cause the processors to display, via the graphical user interface, network devices that support device-independent commands selected from one or more device-independent commands, wherein each device-independent command performs one or more operations on the supported network devices; receive, via the graphical user interface, user input that selects two or more of the displayed network devices, wherein each network device has a network device family, wherein one of the two or more selected network devices is from a first device family and responds to a first set of device commands associated with the first network device family, and wherein another of the two or more selected network devices is from a second device family and responds to a second, different set of device commands associated with the second network device family; perform one or more operations of the selected device-independent command on the selected network devices, wherein performing the one or more operations includes: performing a task on each selected network device based on commands from a set of commands associated with the network device family of the respective selected network device, wherein the task, when executed, performs one or more operations on the respective selected network device.

[0011] In another example, a method includes: defining, in a network controller, a device-independent command that, when executed, performs one or more operations on supported network devices; identifying two or more network device families; configuring the device-independent command to execute on the identified network device families, wherein network devices from one of the identified network device families are associated with and respond to a set of commands, and network devices from another of the identified network device families are associated with and respond to a different set of commands, wherein configuring includes: selecting, for each identified network device family and based on the set of commands associated with each identified network device family, one or more tasks that, when executed, perform one or more operations of the device-independent command on network devices from the respective identified network device family; selecting a network device associated with one of the identified network device families; and performing an operation of the device-independent command on the selected network device, wherein performing includes: executing the task selected for the network device family associated with the selected network device.

[0012] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the invention will be apparent from the description, drawings, and claims. Description of the Drawings

[0013] Figure 1 is a block diagram of an example network of a data center illustrating an example of data with which the techniques described herein may be implemented.

[0014] Figure 2 illustrates in more detail Figure 1 an example implementation of a data center.

[0015] Figure 3A is a block diagram of a software-defined network illustrating the techniques described herein.

[0016] Figure 3B is a block diagram of an example method of a virtual network controller according to one or more techniques of the present disclosure. Figure 3A

[0017] Figure 3C illustrates Figure 3A tunnel communication in a software-defined network.

[0018] Figure 4 is a block diagram of a computing device of an example virtual router that executes a virtual network according to the techniques described herein.

[0019] Figure 5 is a flowchart of an example method for executing device-independent commands on a network device of a software-defined network according to the techniques described herein. Figure 3A and Figure 3B

[0020] Figures 6A to 6H illustrates a representative user interface for supporting device-independent commands according to the techniques described herein.

[0021] Figure 7 is a flowchart of an example method for creating new device-independent commands on a network device of a software-defined network according to the techniques described herein. Figure 3A and Figure 3B

[0022] Figure 8 is a block diagram of an example job template associated with device-independent commands according to the techniques described herein.

[0023] Figure 9 illustrates a representative network device on which device-independent commands may be executed according to the techniques described herein.

[0024] Figure 10 illustrates an example directory structure that may be used to organize device-independent commands according to the techniques described herein.​​​

[0025] Figure 11 illustrates Figure 9 an example Ansible script for a router of

[0026] Figure 12 illustrates an example template for device - independent commands on a specified network device according to one aspect of the present disclosure.

[0027] Figure 13 illustrates an output file obtained by executing device - independent commands on a selected network device according to one aspect of the present disclosure.

[0028] Throughout the drawings and the text, like reference numerals represent like elements. Detailed Description

[0029] As noted above, a software - defined data center (SDDC) is a data center that uses software - defined networking (SDN) to virtualize the underlying infrastructure and deliver aspects of the virtualized infrastructure as services. Software - defined data centers typically use an intent model to automate day - to - day tasks for delivering multi - tenant services over a data center (DC) fabric. However, intent - based automation tools do not allow system users to implement specific operations in a traditional element management system (EMS). Currently, a user / administrator may need to compromise between the ease of control provided by SDN abstractions (intents) and the better control provided by low - level operations performed via a command - line interface (CLI) or a traditional EMS. The present disclosure describes techniques that enable an administrator to specify and execute device - independent commands (also described as general device operations) to implement specific operations on devices in a network.

[0030] Figure 1 is a block diagram of an example network 8 of a data center 10 in which the techniques described herein may be implemented. Generally, the data center 10 provides an operating environment for applications and services to customers 11 coupled to the data center via a service - provider network 7. The data center 10 may, for example, host infrastructure devices such as network and storage systems, redundant power supplies, and environmental controls. The service - provider network 7 may be coupled to one or more networks managed by other providers and may thus form part of a large - scale public network infrastructure (e.g., the Internet). In some examples, the data center 10 may be distributed across several geographically - distributed locations.

[0031] As Figure 1As illustrated in the example, the data center 10 can be a facility that provides network services to customers 11. Customers 11 can be collective entities such as enterprises, governments, or individuals. For example, a network data center can host web services for several enterprises and end-users. Other exemplary services can include data storage, virtual private networks, business engineering, file services, data mining, scientific computing, or supercomputing, etc. In some embodiments, the data center 10 can be a single network server, a network peer, and so on.

[0032] In this example, the data center 10 includes a storage system and a collection of application servers 12A - 12X (herein, "servers 12"), which are interconnected via a high-speed switch fabric 14 provided by one or more layers of physical network switches and routers. In Figure 1 the example shown, the switch fabric 14 includes a collection of interconnected top-of-rack (TOR) switches 16A - 16BN (collectively referred to as "TOR switches 16"), which are coupled to the distribution layer of chassis switches 18A - 18M (collectively referred to as "chassis switches 18"). Although not shown, the data center 10 can also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.

[0033] In Figure 1 the example shown, the TOR switches 16 and the chassis switches 18 provide redundant (multi-homed) connections to the servers 12 via subnets 17.1 - 17.N (collectively referred to as "subnets 17") to the IP fabric 20 and the service provider network 7. The chassis switches 18 aggregate traffic flows and provide high-speed connections between the TOR switches 16. The TOR switches 16 can include network devices that provide layer 2 (e.g., MAC) and / or layer 3 (e.g., IP) routing and / or switching functions. Each of the TOR switches 16 and the chassis switches 18 can include one or more processors and memories and can be capable of executing one or more software processes. The chassis switches 18 are coupled to the IP fabric 20, which performs layer 3 routing to route network traffic between the data center 10 and the customers 11 through the service provider network 7.

[0034] According to one or more embodiments of the present disclosure, the virtual network controller 22 (“VN controller 22”) provides a logically and in some cases physically centralized controller for facilitating the operation of one or more virtual networks within the data center 10. In some examples, the virtual network controller 22 may operate in response to configuration inputs received from the network administrator 24. Additional information regarding the virtual network controller 22 operating with other devices of the data center 10 or with other software-defined networks can be found in the international application number PCT / US2013 / 044378, entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS,” filed on Jun. 5, 2013, the description of which is incorporated herein by reference as if set forth in full herein.

[0035] In one example method, the virtual network controller 22 is a logically centralized but physically distributed software-defined network (SDN) controller. Physically distributed means that the virtual network controller 22 may include multiple types of nodes, and each node may have multiple instances for high availability (HA) and horizontal scaling. In one such example method, the virtual network controller 22 includes three types of nodes: a configuration node, a control node, and an analytics node. These node instances may be implemented in physical servers 12 or on virtual machines. In one such example method, the configuration node in the virtual network controller 22 configures the control node via a technology data model stored on a metadata access point interface (IF-MAP) server 26 such as Figure 2 shown. In some example methods, the virtual network controller 22 is implemented using the Contrail controller available from Juniper Networks or using the open-source network virtualization platform Tungsten Fabric available from the Linux Foundation.

[0036] Typically, for example, traffic between any two network devices (such as between network devices within an IP fabric 20 (not shown), or between server 12 and client 11, or between servers 12) can traverse the physical network using many different paths. For example, there may be several different paths of the same cost between two network devices. In some cases, packets belonging to network traffic from one network device to another can be distributed among various possible paths using a routing policy called multipath routing at each network switching node. For example, Internet Engineering Task Force (IETF) RFC 2992, “Analysis of an Equal-Cost Multi-Path Algorithm” describes a routing technique for routing packets along multiple equal-cost paths. The technique of RFC 2992 analyzes a particular multipath routing policy that involves assigning flows to bins by hashing packet header fields, and sending all packets from a particular network flow over a single deterministic path.

[0037] For example, a “flow” can be defined by five values used in the header of a packet, or a “five-tuple” (i.e., the protocol, source IP address, destination IP address, source port, and destination port used to route the packet through the physical network). For example, the protocol specifies the communication protocol, such as TCP or UDP, and the source and destination ports refer to the source and destination ports of the connection. A set of one or more packet data units (PDUs) that match a particular flow entry represents a flow. Any parameter of the PDU can be used to broadly classify the flow, such as the source and destination data link (e.g., MAC) and network (e.g., IP) addresses, virtual local area network (VLAN) tag, transport layer information, multi-protocol label switching (MPLS) or generalized MPLS (GMPLS) label, and the ingress port of the network device receiving the flow. For example, a flow can be all PDUs transmitted in a Transmission Control Protocol (TCP) connection, all PDUs provided by a particular MAC address or IP address, all PDUs with the same VLAN tag, or all PDUs received at the same switch port.

[0038] In accordance with various aspects of the techniques described in this disclosure, one or more servers in server 12 may include virtual routers that execute multiple routing instances for corresponding virtual networks within data center 10. Packets received by the virtual router of server 12A, for example, from the underlying physical network fabric, may include an outer header to allow the physical network fabric to tunnel the payload or "inner packet" to the physical network address of the network interface of server 12A that executes the virtual router. The outer header may include not only the physical network address of the network interface of the server, but also a virtual network identifier, such as a VxLAN tag or a Multiprotocol Label Switching (MPLS) tag, that identifies one of the virtual networks within the virtual network and the corresponding routing instance executed by the virtual router. The inner packet includes an inner header that has a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier. In Figure 1 the example method, the virtual router (vRouter) 30 on server 12A communicates via tunnel 15 with vRouter 30 on server 12X, as will be discussed in more detail below.

[0039] In some example methods, the vRouters 30 aggregate tunneled packets received from the underlying physical network fabric before delivering the packets to the appropriate routing instance for each packet. In some examples, the vRouter 30 aggregates tunneled packets according to a matching criterion that includes the virtual network identifier of the outer header and one or more fields of the inner header. That is, the vRouter 30 executed on one of the servers in server 12 may receive inbound tunnel packets of a packet flow from switch 16 and, before routing the tunnel packets to a locally executed virtual machine, process the tunnel packets to construct a single aggregated tunnel packet for forwarding to the virtual machine. That is, the vRouter 30 may buffer multiple inbound tunnel packets and construct a single tunnel packet, where the payloads of the multiple tunnel packets are combined into a single payload, and the outer / overlay headers on the tunnel packets are removed and replaced with a single header virtual network identifier. Thus, the aggregated tunnel packet may be forwarded by the vRouter 30 to the virtual machine as if a single inbound tunnel packet had been received from the virtual network. Moreover, to perform the aggregation operation, the vRouter 30 may utilize a kernel-based offload engine that seamlessly and automatically directs the aggregation of the tunnel packets.

[0040] In some example methods, the vRouter 30 executing on the server 12 can manipulate the inbound tunnel packets received among multiple processor cores to facilitate packet processing load balancing among the cores when processing the packets for routing to one or more virtual machines and / or physical machines. As an example, the server 12A can include multiple network interface cards and multiple processor cores to execute the virtual router, and can manipulate the received packets among the multiple processor cores to facilitate packet processing load balancing among the cores. For example, a particular network interface card of the server 12A can be associated with a designated processor core, and the network interface card indicates all received packets to the designated processor core. According to a hash function applied to at least one of the inner packet header and the outer packet header, instead of each of the received packets being processed by various processor cores, a flow may be offloaded to one or more other processor cores for processing to utilize the available work cycles of the other processor cores.

[0041] In some such example methods, the virtual router 30 executing on the server 12 can actively add flow table entries via the virtual router 30 to identify the reverse flows of the flows processed by the routing instances of the virtual router 30. In an example implementation, the virtual router of the server 12A can actively add flow table entries to identify the reverse flows of the flows processed by the routing instances of the virtual router. For example, the virtual machines executing on the server 12A and the members of the virtual network implemented by the data center 10 can receive the initial inbound tunnel packets of the packet flows initiated by the virtual machines executing on the server 12X and the members of the virtual network. When receiving the initial inbound tunnel packets, in addition to adding flow table entries specifically for the inbound packet flow, the virtual router of the server 12A can also actively add flow table entries specifically for the reverse packet flow (i.e., the outbound packet flow) that corresponds to the received inbound packet flow. In this way, the server 12A can predict the need to process the outbound tunnel packets with reverse flow criteria, and thus more effectively look up and use the flow table entries of the reverse packet flow to process the subsequent packets belonging to the reverse packet flow. The approach described above is also described in U.S. Patent No. 9,571,394, entitled “TUNNELED PACKET AGGREGATION for VIRTUAL NETWORKS,” which was authorized on February 14, 2017, and is incorporated herein by reference in its entirety. Figure 1 in the context described above.

[0042] Figure 2 is a block diagram that more particularly illustrates Figure 1 an example implementation of the data center 10. In Figure 2In the example, data center 10 includes an overlay network that extends switch fabric 14 from physical switches 16, 18 to software or “virtual” switches 30A - 30X (collectively “vRouter 30”). VRouter 30 dynamically creates and manages one or more virtual networks 34 that can be used for communication between application instances. In one example, vRouter 30 implements the virtual network as an overlay network that provides the ability to decouple the virtual addresses of applications from the physical addresses (e.g., IP addresses) of the servers 12 on which the applications are executing. Thus, each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and across virtual network 34 over the physical network. For example, virtual network 34 can be configured to provide multicast services within the virtual network without requiring multicast support in the underlying physical network.

[0043] Each vRouter 30 can execute within a hypervisor 31, the host operating system, or other components of each server in server 12. Each server in server 12 can represent an x86 or other general - purpose or special - purpose server capable of executing virtual machines 36. In Figure 2 the example, vRouter 30A executes within hypervisor 31, which is also commonly referred to as a virtual machine monitor (VMM), and which provides a virtualization platform that allows multiple operating systems to run concurrently on one of the servers in server 12. In Figure 2 the example, vRouter 30A manages virtual network 34, and each virtual network 34 provides a network environment for use by one or more virtual machines (VMs) 36 executing on the virtualization platform provided by hypervisor 31. Each VM 36 is associated with one of the virtual networks VN0 - VN2 and can represent a tenant VM running a customer application, such as a web server, a database server, an enterprise application, or a hosted virtualization service for creating a service chain. In some cases, any one or more of the servers in server 12 or another computing device can directly host a customer application, i.e., not as a virtual machine. The virtual machines (e.g., VM 36, 110) and the servers 12 or another separate computing device hosting a customer application referred to herein can alternatively be called “hosts”.

[0044] Generally, each VM 36 can be any type of software application and can be assigned a virtual address for use within a corresponding virtual network 34, where each virtual network in the virtual network can be a different virtual subnet provided by vRouter 30A. The VM 36 can be assigned its own virtual layer 3 (L3) IP address, e.g., for sending and receiving communications, but may not know the IP address of the physical server 12A on which the virtual machine is executing. Thus, a "virtual address" is the address of an application, which is different from the logical address of the underlying physical computer system (e.g., Figure 2 the server 12A in the example of

[0045] In one implementation, each server 12 includes a corresponding one of virtual network (VN) agents 35A - 35X (collectively referred to as "VN agents 35") that control the overlay of the virtual network 34 and coordinate the routing of data packets within the server 12. Generally, each VN agent 35 communicates with a virtual network controller 22 that generates commands to control the routing of packets through the data center 10. The VN agent 35 can operate as a proxy for controlling control plane messages between the virtual machine 36 and the virtual network controller 22. For example, the VM 36 can request to send a message using its virtual address via the VN agent 35A, and the VN agent 35A can in turn send the message and request to receive a response to the message addressed to the virtual address of the VM 36 that initiated the first message. In some cases, the VM 36 can initiate a procedure or function call presented by the application programming interface of the VN agent 35A, and the VN agent 35A can also handle the encapsulation of the message, which includes addressing.

[0046] In one example, a network packet (e.g., a Layer 3 (L3) IP packet or a Layer 2 (L2) Ethernet packet) generated or consumed by an instance of an application executed by a virtual machine 36 within a virtual network domain can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted by a physical network. Packets transmitted within the virtual network can be referred to herein as “inner packets,” while physical network packets can be referred to herein as “outer packets” or “tunnel packets.” Encapsulation and / or decapsulation of the virtual network packet within the physical network packet can be performed within the vRouter 30 (e.g., within the hypervisor 31 or on the host operating system running on each server in the server 12). As another example, the encapsulation and decapsulation functions can be performed at the edge of the fabric 14 at the first-hop TOR switch 16, which is one hop removed from the application instance originating the packet. As noted above, this function is referred to herein as tunneling and can be used within the data center 10 to create one or more overlay networks. Other example tunneling protocols that can be used in addition to IPinIP include IP over GRE, VxLAN, MPLS over GRE, MPLS over UDP, etc.

[0047] As noted above, the virtual network controller 22 provides a logically centralized controller for facilitating the operation of one or more virtual networks within the data center 10. The virtual network controller 22 can, for example, maintain a routing information base, e.g., one or more routing tables storing routing information for the physical network as well as for one or more overlay networks of the data center 10. Similarly, the switches 16, 18, and the vRouter 30 can maintain routing information, such as one or more routing and / or forwarding tables. In one example implementation, the vRouter30A of the hypervisor 31 implements a network forwarding table (NFT) 32 for each virtual network 34. Generally, each NFT 32 stores forwarding information for the corresponding virtual network 34 and identifies where the data packet is to be forwarded and whether the packet is to be encapsulated in a tunneling protocol, such as using a tunnel header that can include one or more headers for different layers of the virtual network protocol stack.

[0048] In one such example method, the virtual machine VM1 sends an “inner packet” to the vRouter30A via an internal link. The VRouter 30A uses the NFT1 to look up the virtual network destination network address of the inner packet. In one such example method, the NFT1 specifies the outbound interface of the vRouter30A and the encapsulation of the inner packet. The VRouter 30A applies the encapsulation to add a tunnel header to generate an outer packet and outputs the outer packet on the outbound interface, or in this case, outputs the outer packet towards the TOR switch 16A.

[0049] For example, the routing information can map the group key information (e.g., destination IP information and other selected information from the packet header) to one or more specific next hops within the network provided by the vRouters 30 and the switch fabric 14. In some cases, the next hop can be a link next hop that specifies the set of operations to be performed on each packet when forwarding the packet, such as can be used for flooding the next hop and multicast replication. In some cases, the virtual network controller 22 maintains the routing information in the form of a radix tree having leaf nodes representing destinations within the network. U.S. Patent 7,184,437 provides details of an exemplary embodiment of a router that utilizes a radix tree for route resolution, the description of which is incorporated herein by reference in its entirety.

[0050] As Figure 2 shown, each virtual network 34 provides an encapsulated packet communication framework 37 for the overlay network established through the switch fabric 14. In this way, the encapsulated packet communication framework 37 can be used to transmit network packets associated with any of the virtual machines 36 via the overlay network. Additionally, in Figure 2 the example of, each virtual router (vRouter) 30 includes a default network forwarding table NFT0 and provides a default route that allows packets to be forwarded to the virtual subnet VN0 without encapsulation, i.e., unencapsulated packet communication 39 according to the routing rules of the physical network of the data center 10. In this way, the subnet VN0 and the virtual default network forwarding table NFT0 provide a mechanism for bypassing the overlay network and sending unencapsulated packet communication to the switch fabric 14 via the unencapsulated communication framework 39.

[0051] Moreover, the virtual network controller 22 and the vRouter 30 can communicate using the virtual network VN0 as a subnet according to the default network forwarding table NFT0 32 during the discovery and initialization of the overlay network and during conditions where a failed link has temporarily stopped communication via the overlay network. Once a connection to the virtual network controller 22 is established, the virtual network controller 22 can update its local routing table to accommodate new information about any failed links and can instruct the vRouter 30 to update its local network forwarding table 32. For example, the virtual network controller 22 can output commands to the virtual network agent 35 to update one or more NFTs 32 to instruct the vRouter 30 to change the tunnel transport encapsulation in order to reroute communication within the overlay network, e.g., to avoid a failed link.

[0052] When a link failure is detected, the virtual network agent 35 local to the failed link (e.g., VN agent 35A) can immediately change the encapsulation of network packets to redirect traffic within the overlay network and can notify the virtual network controller 22 of the routing change. In turn, the virtual network controller 22 can update its routing information and can send messages to other virtual network agents 35 to update the local routing information stored by the virtual network agents within the network forwarding table 32.

[0053] Figure 3A is a block diagram illustrating an example software-defined network implementation of a network 8( Figures 1 to 2 ) according to the techniques described herein. In one example method, each vRouter 30 forwards packets from one virtual machine 36 to other virtual machines via a set of server-to-server tunnels. The tunnels form an overlay network on top of the physical network (such as, for example, a physical IP-over-Ethernet network). In Figure 3A the example shown, virtual machines 36 on one vRouter 30 communicate with virtual machines 36 on other vRouter 30s via MPLS over GRE, MPLS over UDP, or VXLAN.

[0054] In Figure 3A the example method, the virtual network controller 22 is a software-defined network (SDN) controller. As noted in the discussion above in Figure 1 and Figure 2 , in some example methods, the virtual network controller 22 is logically centralized but can be physically distributed across many devices. In one example method, the controller 22 includes a user interface 45 and various types of nodes, each of which can have multiple instances for high availability (HA) and horizontal scaling. In one example method, such as Figure 3A shown in Figure 2 , the virtual network controller 22 includes three types of nodes: a configuration node 40, a control node 42, and an analytics node 44. These node instances can be implemented in physical servers 12 or on virtual machines 36. In one such example method, the configuration node 40 configures the control node 42 via a technology data model stored on the

[0055] In one example method, the configuration node 40 provides a management layer for configuring the control node 42. In Figure 3AIn the example shown, the configuration node 40 provides a northbound Representational State Transfer (REST) Application Programming Interface (API) 43 that can be used by, for example, an orchestration program 46 or an automation interface 48 to configure the network 8 or extract the operational state of the network 8. In one example method, an instantiated service is represented by an object in a horizontally scalable database described by a formal service data model. The configuration node 40 also includes a transformation engine (sometimes referred to as a compiler) that transforms objects in a high-level service data model into corresponding lower-level objects in a technical data model. In some example methods, the transformation engine can be accessed via a user interface 45. An example transformation engine is described in U.S. Patent Application No. 15 / 198,657, filed Jun. 30, 2016, entitled “TRANSLATING HIGH-LEVEL CONFIGURATION INSTRUCTIONS TO LOW-LEVEL DEVICE CONFIGURATION,” the description of which is incorporated herein by reference.

[0056] In one example method, the control node 42 implements a logically centralized portion of the control plane. In one example method, not all control plane functions are logically centralized, and some control plane functions are still implemented in a distributed manner on physical and virtual routers and switches in the network 8. The control node 42 can use the IF-MAP protocol to monitor the content of the low-level technical data model computed by the configuration node 40. The low-level data model describes the desired state of the network. The control node 42 can use a combination of southbound protocols (such as the Extensible Messaging and Presence Protocol (XMPP)) to configure the vRouter 30 and use the Border Gateway Protocol (BGP) and Network Configuration (NETCONF) protocols to control physical routers (such as the underlying switch 50 of the IP fabric 20). In some such example methods, the BGP and NETCONF protocols can also be used to control the gateway 52. In one example method, when there are multiple instances of the control node 42, the control node 42 also uses BGP for state synchronization between each other, such as for reasons of scale-out and high availability (HA).

[0057] In one example method, objects that describe an instantiated service are defined via a formal service data model. In such an example method, the formal service data model is a high-level service data model that is used to describe the service that needs to be implemented; each high-level service data model has an associated low-level technical data model that describes how these services need to be implemented. Each data model consists of a collection of objects, their capabilities, and the relationships between them.

[0058] In an example method, each formal service data model can be transformed into a low-level technical data model that describes how to implement the service. In one such example method, the configuration node 40 in the virtual network controller 22 transforms any changes in the high-level service data model into a corresponding set of changes in the low-level technical data model. The configuration node 40 can then use the IF-MAP protocol to publish the content of the low-level technical data model stored on the Metadata Access Point Interface (IF-MAP) server 26 to the control node 42.

[0059] In an example method, the analysis node 44 of the VN controller 22 is used to collect, collate, and present analysis information for troubleshooting problems and for determining network usage. In one such example method, each component of the network 8 generates detailed event records of important events in the system. These event records can be sent to one of multiple instances (for scale-out) of the analysis node 44, which uses a format optimized for time series analysis and querying to collate and store the information in a horizontally scalable database. The analysis node 44 can also include mechanisms that automatically trigger the collection of more detailed records when certain events occur, allowing the VN controller 22 to find the root cause of any problem without having to reproduce it. In an example method, the analysis node 44 also provides a northbound analytics query REST API that can be used by an orchestrator 46 or an automation interface 48 to retrieve analytics.

[0060] In an example method, the VN controller 22 is physically distributed but logically centralized. In one such example method, the VN controller 22 supports multiple redundant instances of any node. The redundant instances operate in an active-active mode (as opposed to an active-standby mode). By operating in an active-active mode, the VN controller 22 can continue to operate without any interruption when any given node fails. Further, when a node becomes overloaded, additional instances of that node type can be instantiated and the load automatically redistributed. This prevents any single node from becoming a bottleneck and allows the system management to support very large systems with tens of thousands of servers. Logical centralization means that the VN controller 22 behaves as a single logical unit, even though it is implemented as a physically distributed cluster of multiple nodes.

[0061] In an example method, the configuration node 40 provides a discovery service that clients can use to locate service providers (i.e., other nodes that provide a specific service). For example, when the vRouter agent 35 in the compute node 12 wants to connect to the control node 42 (or, in some cases, to an active-active pair of control VMs 42), it uses service discovery to discover the IP address of the control node 42. In some such example methods, clients use local configuration, DHCP, or DNS to locate the service discovery server.

[0062] In an example method, the virtual network controller 22 implements three basic building blocks: multi-tenancy, gateway functionality, and service chaining. Multi-tenancy (also known as network virtualization or network slicing) is the ability to create a virtual network that provides a closed user group for a collection of VMs 36. Gateway functionality refers to the ability to connect the virtual network to the physical network via a gateway router (e.g., the Internet) and to attach non-virtualized servers or network services to the virtual network via a gateway (such as Figure 3A gateway 52). Service chaining (also known as NFV) is the ability to manipulate traffic flows through a sequence of physical or virtual network services (such as firewalls, DPI, or load balancers).

[0063] In an example method, the physical routers and switches of the underlying network do not contain any per-tenant state. That is, they do not contain the media access control (MAC) addresses, IP addresses, or policies of the virtual machines 36. The forwarding tables of the underlying physical routers and switches only contain the IP prefixes or MAC addresses of the physical servers 12. However, the gateway routers or switches that connect the virtual network to the physical network do contain tenant MAC or IP addresses.

[0064] In one such example method, the vRouters 30 do contain per-tenant state. They contain separate forwarding tables (routing instances) for each virtual network. This forwarding table contains the IP prefixes (in the case of layer 3 overlay) or MAC addresses (in the case of layer 2 overlay) of the virtual machines 36. A single vRouter 30 does not need to contain all of the IP prefixes or all of the MAC addresses of all of the virtual machines 36 in the entire data center 10. A given vRouter 30 may only need to contain those routing instances that are locally present on the server 12 (i.e., it has at least one virtual machine present on the server 12).

[0065] In some example methods, the orchestrator 46 automates the provisioning of services in a software-defined network. In some such example methods, the orchestrator 46 can, for example, receive a request from a customer 11 to add and provision a new virtual server, analyze the network configuration, and then implement the configuration change via the VN controller 22. The orchestrator 46 can also update the billing system, which records the services added for the customer. Representative orchestrators include Openstack, Kubernetes, Openshift, and vCenter.

[0066] In some example methods, the automation interface 48 uses script code and other automation tools to reduce the operational workload of configuring, troubleshooting, and managing the physical and virtual devices of the system 10. In some such example methods, the automation takes the form of Python or Perl script code, which is used to perform a specific or small set of functions. In other such example methods, more sophisticated automation tools such as Ansible and Puppet can be used. Ansible is a provisioning, configuration, and deployment tool that relies on scripts to define sequences of tasks. Puppet is a configuration management tool that is used to simplify the definition of IT infrastructure as code and the implementation of system configurations. In one example method, the VN controller 22 includes its own automation interface (similar to, for example, the automation interface 48), which can be accessed via the user interface 45. In one such example method, the user interface 45 is a graphical user interface (GUI) built using the REST API 43 to communicate with the automation interface of the VN controller 22.

[0067] Figure 3B is a block diagram illustrating an example method of a Figure 3A virtual network controller in accordance with one or more techniques of the present disclosure. In Figure 3B the example shown, the virtual network controller 22 includes one or more configuration nodes 40, one or more control nodes 42, one or more analysis nodes 44, and an interface unit 60 connected to the user interface 45. Node instances 40, 42, 44 can be implemented in a physical server 12 or on a virtual machine 36. In one such example method, the configuration node 40 configures the control node 42 in the virtual network controller 22 via a technology data model stored on the Figure 2 metadata access point interface (IF-MAP) server 26 shown in

[0068] In some examples, the virtual network controller 22 can include one or more controller devices that are included in a network, where the one or more controller devices individually and / or collectively include at least one processor, and the at least one processor includes processing circuitry ( Figure 3B(not shown in the figure). In some cases, the processing circuit may execute software instructions, such as software instructions for defining one or more software or computer programs, which are stored in a computer-readable storage medium ( Figure 3B (not shown in the figure), such as a non-transitory computer-readable medium, which includes a storage device or memory that stores instructions for causing one or more processors to execute the techniques described herein. Alternatively or additionally, at least one processor may include dedicated hardware for executing the techniques described herein (e.g., one or more integrated circuits, one or more application-specific integrated circuits (ASICs), one or more application-specific special processors (ASSPs), one or more field-programmable gate arrays (FPGAs), or any combination of one or more of the foregoing examples of dedicated hardware).

[0069] As Figure 3B shown, the virtual network controller 22 includes an interface unit 60, one or more analysis nodes 44, one or more configuration nodes 40, and one or more control nodes 42. Generally, each of the interface unit 60, the analysis nodes 44, the configuration nodes 40, and the control nodes 42 may be implemented as software, hardware, firmware, or any combination thereof, and may be distributed across one or more hardware computing platforms that provide an environment for the implementation of these units (e.g., distributed across one or more control devices in a network). Moreover, each of the interface unit 60, the analysis nodes 44, the configuration nodes 40, and the control nodes 42 may maintain state data, which may be stored in a centralized or distributed database.

[0070] For example, as Figure 3B shown, the virtual network controller 22 includes various data memories or databases, as pointed out above, which may be stored centrally or distributed across the nodes of the virtual network controller 22. These data memories include a data model 80, topology information 82, device status information 84, configuration information 86, and routing information 88. The analysis nodes 44, the configuration nodes 40, the interface unit 60, and the control node 40 are all communicatively coupled to the data model 80, the topology information 82, the device status information 84, the configuration information 86, and the routing information 88. In one example method, the data memories further include a job template 90 and user-defined commands / patterns 92. The analysis nodes 44, the configuration nodes 40, the interface unit 60, and the control node 40 are all communicatively coupled to include the job template 90 and the user-defined commands / patterns 92. In one example method, each user-defined device-independent command includes one or more of the following: an input mode, an output mode, and an input_ui mode, as further defined below.

[0071] In some example methods, the interface unit 60 uses script code and other automation tools to reduce the operational workload of configuring, troubleshooting, and managing the physical and virtual devices of the system 10. In some such example methods, the automation takes the form of Python or Perl script code, which is used to perform a specific or small set of functions. In other such example methods, more complex automation tools such as Ansible and Puppet can be used. In one example method, the VN controller 22 includes a command processing unit 62, which is used to compile and execute user-defined device-independent commands via the configuration node 40 or the analysis node 44. In some such example methods, the command presentation unit 64 operates in conjunction with the command processing unit 62 to present a framework to the user for defining and executing such user-defined device-independent commands. In one such example method, the user interface 45 is a graphical user interface (GUI) built using the REST API 43 to communicate with the command presentation unit 64 of the VN controller 22.

[0072] In one example method, the command presentation unit 64 presents a data model-driven framework, which is reproduced (without any hard coding) through the UI 45 for the user to initiate general (device-independent) operations on a set of devices. In one such example method, the framework uses standard data models and open-source tools such as Ansible and Jinja2 (Jinja2 is a templating language for Python), and leverages the following capabilities of the VN controller 22 and the UI 45: reproducing general operations that the user can customize and execute on the selected network devices via the command presentation unit 64, receiving results via the command presentation unit 64, and reproducing the results on the UI 45 through a data model-driven approach.

[0073] In an example method, a user interface 60 displays, via a user interface 45, network devices that support device-independent commands selected from one or more device-independent commands, where each device-independent command performs one or more operations on a supported network device; receives, via the user interface 45, user input that selects two or more of the displayed network devices, where each network device has a network device family, where one of the two or more selected network devices is from a first device family and responds to a first set of device commands associated with the first network device family, and where another of the two or more selected network devices is from a second device family and responds to a second, different set of device commands associated with the second network device family; and performs one or more operations of the selected device-independent command on the selected network devices, where performing the one or more operations includes: performing a task on each selected network device based on commands from a set of commands associated with the network device family of the respective selected network device, where the task, when executed, performs one or more operations on the respective selected network device. In an example method, the set of commands is a set of command-line interface (CLI) commands associated with a device family and / or a device vendor.

[0074] In an example method, a command processing unit 62 reproduces a user-defined device-independent command and forwards instructions to one of a configuration node 40 and an analysis node 44 for execution. In one such example method, results are returned to the command processing unit 64 for presentation to the user.

[0075] This idea completely eliminates the need for device-specific operations / commands and their associated user interface dependencies. Generic operations rely on input and output data models dynamically reproduced via the command processing unit 62. Moreover, such operations utilize the concepts of role-based and profile-based device management to decouple configuration and operation functions from hard-coded software. Thus, this framework can be used to automate a single operation across multiple device vendors, providing a consistent view of the results to the user (regardless of the vendor-specific output / data provided by each vendor's device model).

[0076] In some example methods, the configuration node 40 includes one or more device managers 70 that provide a management layer for configuring the control node 42. In some example methods, instantiated services are represented by objects in a horizontally scalable database described by a formal service data model stored in a data model memory 80. In some example methods, the device manager 70 also includes the above in Figure 3AThe transformation engine described in the discussion above. As noted above, the transformation engine can be used to transform objects in a high-level service data model into corresponding lower-level objects in a technical data model.

[0077] In one example method, the control node 42 implements the logically centralized portion of the control plane. In one example method, objects that instantiate services are defined via a formal service data model. In such an example method, the formal service data model is a high-level service data model for describing the services that need to be implemented; each high-level service data model has an associated lower-level technical data model that describes how these services need to be implemented. Each data model consists of a collection of objects, their capabilities, and the relationships between them. The data models are stored in the data model memory 80.

[0078] In one example method, each formal service data model can be transformed into a lower-level technical data model that describes how the services are to be implemented. In such an example method, the configuration node 40 transforms any changes in the high-level service data model into a corresponding set of changes in the lower-level technical data model in the virtual network controller 22. Then, the configuration node 40 can use the IF-MAP protocol to publish the content of the lower-level technical data model stored on the Metadata Access Point Interface (IF-MAP) server 26 to the control node 42.

[0079] In some example methods, the configuration node 40 also includes an intent and policy engine (not shown) that is configured to determine the network state, compare the current network state with a desired network state, and drive configuration changes to the network devices 12, 16, 18 to achieve the desired network state. An example intent and policy engine is described in U.S. Patent Application No. 16 / 221,698, titled "NETWORK DEVICE CONFIGURATION USING A MESSAGE BUS," filed on December 17, 2018, the description of which is incorporated herein by reference.

[0080] In some example methods, the configuration node 40 also includes a maintenance mode controller (not shown). An example maintenance mode controller is described in U.S. Patent Application No. 16 / 230,156, titled "AUTOMATION OF MAINTENANCE MODE OPERATIONS FOR NETWORK DEVICES," filed on December 21, 2018, the description of which is incorporated herein by reference.

[0081] Generally, the task of analytics node 44 is to collect, store, correlate, and analyze information from virtual and physical network elements and / or devices within a data center (e.g., data center 10). This information can include statistics, logs, events, and / or errors for managing routing and network configurations of the data center. Analytics node 44 can store this information in one or more of topology information 82, device status information 84, configuration information 86, and / or routing information 88. Interface unit 60 can be configured to provide a communication interface to one or more entities external to virtual network controller 22, such as to administrator 24( Figure 1 ), user interface 45( Figure 3A ), automation interface 48( Figure 3A ), and / or orchestrator 46( Figure 3A ). In some examples, analytics node 44 can provide the collected information and / or any information stored in topology information 82, device status information 84, configuration information 86, or routing information 88 to interface unit 60, which can output such information to one or more external entities, such as administrator 24 or orchestrator 46.

[0082] In some examples, interface unit 60 can provide any of this information to administrator 24 via a portal application, which can be included in or coupled to interface unit 60. The portal application can provide user interface functionality through which a user can provide input to and receive output from the portal application. For example, interface unit 60 can output logs and / or status information to the user via the portal application such that the user can be notified of such information (e.g., before, during, and / or after performing a particular operation).

[0083] As Figure 3B shown, analytics node 44 includes device / role discovery unit 74 and topology discovery unit 76. Topology discovery unit 76 can be configured to collect, store, correlate, and analyze topology information from the network and fabric, which can be stored in topology information 82. For example, referring to Figure 1 's example, topology discovery unit 76 can collect and determine topology information associated with the switch fabric 14 of network 8 in data center 10, such as the specific topology of chassis switches 18 and TOR switches 16, and this information can be stored in topology information 82. Over time, as network devices 12, 16, 18 are added to or removed from the corresponding network, topology discovery unit 76 operates to determine updated topology information and / or changes, which can be stored in topology information 82.

[0084] The device / role discovery unit 74 may be configured to collect or retrieve information from specific network devices 12, 16, 18 in the network over a given period of time, as well as the roles of these devices. As noted above, over time, individual network devices may be added to or removed from the network (e.g., Figure 1 of network 10). The device / role discovery unit 74 is configured to identify whether a device has been added or removed, and to identify device information associated with these devices. The device / role discovery unit 74 may store such information in the topology information 82 and / or the device status information 84. The device / role discovery unit 74 may also store device role information (e.g., whether a device is a spine or leaf device, whether the device is a chassis switch or a TOR switch or a router, etc.) in the topology information 82 and / or the device status information 84.

[0085] The configuration node 40 of the virtual network controller 22 may be configured to configure one or more of the network devices within the network (e.g., Figure 1 of network 100 and / or Figure 2 of network 200). When the configuration node 40 configures a network device, the configuration node 40 may access any of the data model 80, the topology information 82, the device status information 84, the configuration information 86, and / or the routing information 334. The configuration node 40 may also store any information, including configuration information, in any of these data memories. In some examples, the configuration node 40 presents a northbound application programming interface (API) that interfaces, such as via the interface unit 60, with the command engine 213 ( Figure 2 ).

[0086] In certain examples, the configuration node 40 may selectively configure the fabric (e.g., based on network topology and / or status) in advance (such as when the network devices within the fabric (e.g., Figure 1 of the fabric 14) are initially placed within the management scope of the network controller 22). At this preliminary stage, the configuration node 40 may inject certain configurations (e.g., a combination of an underlying routing protocol policy and an overlay routing protocol policy) into the network devices. In some cases, a specific standard protocol extension (e.g., AS-PATH in the case of the underlying Border Gateway Protocol (BGP)) is configured and kept temporarily inactive, e.g., as part of the underlying configuration of these devices. Then, before starting to perform maintenance mode operations on the network devices, the configuration node 40 may activate the configurations previously injected on these devices, thereby allowing traffic to be diverted from the devices undergoing such operations (e.g., software upgrade). In various examples, the configuration information may include routing instances and / or forwarding policy information.

[0087] As discussed above, the virtual network controller 22 includes one or more control nodes 42. The control nodes 42 may implement a logically centralized control plane responsible for maintaining network state. The control nodes 42 interact with network elements, such as Figure 1 the network devices shown in, to ensure that the network state is ultimately consistent with the desired state specified by the orchestration engine (e.g., orchestration engine 213). In some examples, the control nodes 42 receive configuration state information of the virtual network controller 22 from the device configuration unit 338. Further, the control nodes 42 exchange routes with the VN agents (e.g., VN agent 35 on server 12, as shown in Figure 2 via XMPP. The control nodes 42 also communicate configuration state information (such as routing instances and forwarding policies) to the VN agents (e.g., VN agent 35) via, for example, XMPP for installation within the corresponding virtual routers 30. In some examples, the control nodes 42 may proxy traffic on behalf of a server (e.g., the server 12 shown in Figure 2 ). These proxy requests may be received via XMPP. XMPP is further described in detail in P. Saint-Andre, Extensible Messaging and Presence Protocol (XMPP): Core, IETF RFC6120, March 2011, the entire content of which is incorporated herein by reference.

[0088] Further, the control nodes 42 may exchange routes with a gateway (e.g., the gateway 52 shown in Figure 3A ) via BGP, and communicate the configuration state of the virtual network controller 22 with network devices in a fabric (e.g., the switch fabric 14 shown in Figure 1 and Figure 2 ) via NETCONF. As described above, the functionality provided by the maintenance mode controller 315 may be part of the configuration node 40, the control nodes 42, or a combination thereof, which may access the configuration information 86.

[0089] As shown in Figure 3B , the control node 72 includes a protocol controller 72 that is capable of controlling the communication between the virtual network controller 22 and other devices, agents, entities, etc. via one or more communication protocols (such as, for example, the XMPP protocol, the NETCONF protocol, the BGP protocol, and / or the IF-MAP protocol, listing some examples).

[0090] In some examples, the control node 42 receives configuration status from the maintenance mode controller 315 using IF-MAP. The control node 42 may include one or more control nodes that use IBGP to exchange routes with other control nodes to ensure that all control nodes have the same network state. As described above, the control node 42 uses XMPP to exchange routes with the VN agent 35 on compute nodes (e.g., server 12). The control node 42 may also use XMPP to send configuration status such as routing instances and forwarding policies.

[0091] The control node 42 also exchanges BGP messages with BGP peers, which include any network device that is configured to communicate via BGP and also includes any gateway node (e.g., Figure 3A gateway 52 shown in Figure 2 ). The protocol controller 72 may store routing information associated with any device in the network, which includes compute nodes (e.g.,

[0092] Figure 3C illustrates Figure 3A and Figure 3B a block diagram of the communication in a software-defined network. In one example method, the virtual network controller 22 distributes device commands corresponding to device-independent commands initiated as described in the context of Figure 3B to the network devices 12, 16, and 18. Additionally, in an Figure 3C example method, the virtual network controller 22 configures vRouters 30.1 and 30.2 such that the virtual machines 36 connected to vRouter 30.1 communicate with the virtual machines 36 connected to vRouter 30.2. In some example methods, end-to-end encryption is provided via tunneling mechanisms such as MPLS over GRE, MPLS over UDP, or VXLAN. As can be seen from Figure 3C , in one example method, the IP packet 54 transmitted across the virtual fabric 34 is encapsulated using the MPLS over UDP protocol before being transmitted as an encapsulated IP packet 56 across the physical network via tunnel 58.

[0093] In some such example methods, encapsulation is performed as a service on a service chain or via a dedicated encryption device or firewall. Thus, packets can move unencrypted between physical cards within server 12 or across subnet 17 until they reach the node that performs encryption. However, sometimes it is necessary to provide end-to-end encryption to a virtual router as a means of protecting multi-tenant traffic leaving vRouter 30. For example, a tenant may need to run an application while remaining compliant with PCI DSS (Payment Card Industry Data Security Standard). Thus, they may need to use a highly secure encryption algorithm to encrypt the headers and payloads of IP packets. They may also need to have network multi-tenancy with a locked routing table, where there are only specific routes to endpoints governed by a strict policy framework. A method for encapsulating packets before they are transmitted over an underlying network is described in U.S. Patent Application No. 16 / 146,713, the description of which is incorporated herein by reference.

[0094] Figure 4 is a block diagram of a computing device that illustrates an example virtual router for a virtual network implemented in accordance with the techniques described herein. Computing device 100 may represent Figures 1 to 1 any of the servers 12 in -2 or other devices, such as any of the TOR switches 16 in the TOR switch. In Figure 4 the example method, computing device 100 includes a system bus 104 that couples the hardware components of the hardware environment of computing device 100. System bus 104 connects memory 144, network interface cards (NICs) 106A - 106B (collectively referred to as "NIC 106"), storage device 107, and a multi-core computing environment 102 having a plurality of processing cores 108A - 108J (collectively referred to as "processing" cores 108). Network interface card 106 includes interfaces that are configured to exchange packets using the links of the underlying physical network. Multi-core computing environment 102 may include any number of processors and any number of hardware cores, e.g., from 4 to thousands. Each of the processing cores 108 includes an independent execution unit to execute instructions that conform to the instruction set architecture of the core. The processing cores 108 may each be implemented as a separate integrated circuit (IC), or may be combined within one or more multi-core processors (or "multi-core" processors), each multi-core processor implemented using a single IC (i.e., a chip multi-processor).

[0095] Memory 107 represents a computer-readable storage medium that includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the required information and can be accessed by core 108.

[0096] Memory 144 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM) or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the required program code and program data in the form of instructions or data structures and can be accessed by a computer. Memory 144 provides a physical address space composed of addressable memory locations.

[0097] In some examples, memory 144 may present a non-uniform memory access (NUMA) architecture to the multi-core computing environment 102. That is, core 108 may not have equal memory access times to the various storage media that make up memory 144. In certain instances, core 108 may be configured to use a portion of memory 144 that provides the lowest memory latency to the core to reduce the overall memory latency.

[0098] In some instances, the physical address space for the computer-readable storage media may be shared among one or more cores 108 (i.e., shared memory). For example, cores 108A, 108B may be connected via a memory bus (not shown) to one or more DRAM packages, modules, and / or chips (also not shown) that present a physical address space that can be accessed by cores 108A, 108B. Although this physical address space may provide the lowest memory access time to cores 108A, 108B for any part of the portion of memory 144, at least some parts of the remaining portion of memory 144 can be directly accessed by cores 108A, 108B. One or more of cores 108 may also include L1 / L2 / L3 caches or a combination thereof. The respective caches of cores 108 provide the lowest latency memory access to any of the storage media in the storage media.

[0099] Memory 144, network interface cards (NICs) 106A - 106B (collectively referred to as "NIC 106"), storage device 107, and multi - core computing environment 102 provide an operating environment for a software stack that executes a virtual router 120 and one or more virtual machines 110A - 110K (collectively referred to as "virtual machines 110") connected to routing instances 122A - 122F (collectively referred to as "routing instances 122") via tap interfaces 146A - 146K (collectively referred to as "tap interfaces 146"). The virtual machines 110 can represent Figure 2 an example instance of any of the virtual machines in virtual machine 36. The virtual router 120 can represent Figure 1 , Figure 2 , Figure 3A and Figure 3B an example instance of any of the vRouters 30 in vRouter 30. The computing device 100 divides the virtual and / or physical address space provided by main memory 144 and, in the case of virtual memory, by the combination of memory 144 and memory 107 into a user space 111 allocated for running user processes and a kernel space 112 that is protected and generally inaccessible to user processes.

[0100] Memory 144, network interface cards (NICs) 106A - 106B (collectively referred to as "NIC 106"), storage device 107, and multi - core computing environment 102 can also provide an operating environment for an operating system kernel executing in kernel space 112. The operating system kernel can include, for example, Linux, Berkeley Software Distribution (BSD), another Unix - variant kernel, or a Windows server operating system kernel available from Microsoft Corp. The operating system kernel implements an operating system network stack 123 in kernel space 112 as shown in Figure 4 .

[0101] As further explained below, in some example implementations, the kernel space 112 may be configured with multiple network stacks, which may be beneficial when implementing a virtual network over a physical network. For example, as further described, the operating system network stack 123 may represent a first software network stack executing in the kernel space 112, while the virtual router 120 may implement its own corresponding software network stack, where each network stack implements the corresponding functions of the network layer (e.g., layers 1 to 3 of the OSI model). In some examples, the computing device 100 may be configured to use an IPVLAN driver to transfer packets between different network stacks that operate within the same kernel module (e.g., kernel space 112) of the computing device 100. That is, the IPVLAN driver may be installed and configured to operate as a packet transfer having endpoints on different network stacks configured within the same kernel space of the computing device 100.

[0102] In some instances, the computing device 100 may execute a hypervisor to manage virtual machines 110 ( Figure 4 not shown). Figure 2 An example hypervisor 31 is illustrated. Example hypervisors include the Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. In some examples, dedicated hardware programmed with routing information such as a Forwarding Information Base (FIB) 124 may perform aspects of the virtual router 120.

[0103] Eth0 114A and Eth1 114B represent devices according to the software device model and provide device driver software routines for handling packets received / transmitted by the corresponding NIC 106. Packets received by the NIC 106 from the underlying physical network fabric for the virtual network may include an outer header to allow the physical network fabric to tunnel the payload or "inner packet" to the physical network address of one of the NICs 106 in the NIC 106. The outer header may include not only the physical network address but also a virtual network identifier, such as a VxLAN label or a Multiprotocol Label Switching (MPLS) label, which identifies one of the virtual networks in the virtual network and the corresponding routing instance 122. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier. For example, the virtual router forwarding plane 128 may receive, via Eth1, a packet with an outer header from the NIC 106, the outer header including a VxLAN associated with the routing instance 122A in the virtual router forwarding plane 128. The packet may have an inner header having a destination network address that is the destination address of the VM 110A, which accesses the routing instance 122A via the tap interface 146A.

[0104] The virtual router 120 in this example includes a kernel space 112 module: the virtual router forwarding plane 128; and a user space 111 module: the virtual router agent 142. The virtual router forwarding plane 128 performs the "forwarding plane" or packet forwarding function of the virtual router 120, and the virtual router agent 142 performs the "control plane" function of the virtual router 120. The virtual router agent 142 may represent Figure 2 an example instance of any of the VN agents 35 in the VN agent 35.

[0105] The virtual router forwarding plane 128 includes a plurality of routing instances 122A - 122F (collectively referred to as "routing instances 122") for the corresponding virtual network. In Figure 4In the example shown, each routing instance 122 includes a Forwarding Information Base (FIB) 124 and a flow table 126. Although shown as separate data structures within each routing instance 122, the flow table 126 can be a logical table in some instances, implemented as a single table or other associated data structures, where entries of the corresponding flow table 126 can be identified by virtual network identifiers (e.g., VRF identifiers such as VxLAN tags or MPLS labels). The FIB 124 can include a lookup table that maps destination addresses to destination next hops. The destination addresses can include Layer 3 network prefixes or Layer 2 MAC addresses. The flow table 126 can also enable the application of forwarding policies to flows. In one example method, each flow table in the flow table 126 includes flow table entries that match one or more flows that can traverse the virtual router forwarding plane 128 and include a forwarding policy to be applied to the matching flows. In one example method, for instance, the virtual router forwarding plane 128 attempts to match a packet processed by the routing instance 122A with one of the flow table entries in the flow table 126A. In this example, if there is a matching flow table entry for a given packet in the flow table 126A, the virtual router forwarding plane 128 applies the flow action specified in the policy to the packet. This can be referred to as "fast path" packet processing. If there is no matching flow table entry for the packet in the flow table 126A, the packet can represent the initial packet of a new packet flow; in such a case, the virtual router forwarding plane 128 can request the virtual router agent 142 to install a flow table entry for the new packet flow in the flow table 126A via the link 140. This can be referred to as "slow path" packet processing for the initial packet of a packet flow.

[0106] In one example method, the virtual router agent 142 is a user space 111 process executed by the computing device 100. The virtual router agent 142 includes configuration data 134, virtual routing and forwarding instance configuration 136 ("VRF 136"), and a policy table 138 ("policy 138"). In one example method, the virtual router agent 142 exchanges control information with one or more VN controllers 22. The control information can include virtual network routing and low-level configuration status (such as routing instances and forwarding policies) for installation into the configuration data 134, VRF 136, and policy 138. The virtual router agent 142 can also report analytics status, install forwarding status into the FIB 124 of the virtual router forwarding plane 128, and discover VMs 110 and their attributes. As pointed out above, the virtual router agent 142 can further apply slow path packet processing to the first (initial) packet of each new flow traversing the virtual router forwarding plane 128 and can install the corresponding flow entry into the flow table 126 of the new flow for fast path processing of subsequent packets of the flow by the virtual router forwarding plane 128.

[0107] In an example method, as noted above, the virtual router 120 is a kernel module loaded in the kernel 112, and the host operating system loads the IP framework for transforming packets of the IPSec module 125 at startup. In one such example method, the IPSec module 125 implements IPSec with IKEv2, authentication management for IKEv2, AES-GCM 256 encryption, and AES-NI. In a Linus-based method, the operating system loads XFRM (the Linux IP framework for transforming packets) at startup. In an example method, IPSec is configured in a full-mesh tunnel mode across the network 8 that connects the virtual routers 120 to each other.

[0108] In an example method, the overlay IP packets on the transmit (TX) path are sent from the virtual router 120 to the IPSec module 125 for encryption. Then, the IP packet with ESP (Encapsulating Security Payload) is returned to the virtual router 120 for forwarding by the virtual router.

[0109] In an example method, the virtual router 120 creates IP / MPLS or VXLAN packets for tenant applications with the appropriate L2 and L3 headers of the source and destination, and writes the packets to the IPSec module 125 via the IPSec interface 127. The IPSec kernel executed in the IPSec module 125 captures the packets based on the state and policies provided as part of the IPSec module that enables the connection. Based on the policy matching the IP address in the packet, XFRM transforms and encrypts the entire IP packet using ESP (Encapsulating Security Payload), thus ensuring the authentication, integrity, and confidentiality of tenant traffic. Once the IPSec kernel encrypts the packet, the IPSec module 125 transmits the packet with the IPSec Encapsulating Security Payload (ESP) to the virtual router 120. The virtual router 120 receives the encapsulated packet and transmits the packet to the physical network.

[0110] At the receiving end, the encrypted packet arrives at the destination virtual router 120. In one example method, the ESP packet on the receive (RX) side of the virtual router 120 is sent to the OS network stack 123, where the Linux kernel configured with IPSec decrypts the packet and writes the resulting MPLS / VxLAN packet to the virtual router 120. In one example method, the virtual router 120 is configured to look only for MPLS, VXLAN, and GRE headers. All other packets (including ESP packets) are forwarded to the IPSec module 125 via the IPSec kernel interface 127. In one example method, the IPSec module 125 reads the packet from the interface, decrypts the encrypted packet, and forwards the decrypted IP packet to the virtual router 120. In one example method, the IPSec kernel interface 127 includes a decryption interface detected on the OS network stack and the virtual router stack 120 such that the decrypted packet (substantially an IP / MPLS or VXLAN packet) is read by the virtual router 120 and sent to the appropriate tenant application interface based on label lookups. As noted above, packet processing occurs in the kernel of the IPSec module 125 and the virtual router 120.

[0111] In one example method, the virtual network controller 122 includes an application programming interface (API) for enabling encryption on a per virtual router 120 or per virtual router instance 122 basis. For example, all packets from tenant workloads traversing the virtual router instance 122 can be configured to have encryption enabled so that the packets are encrypted. In one such example method, the virtual network controller 122 provides a graphical user interface (GUI) that allows an administrator to enable or disable encryption for secure forwarding of tenant workloads on a per virtual router instance basis.

[0112] Figure 5 illustrates a method for Figure 3A and Figure 3BFlowchart of an example method for executing device-agnostic commands on network devices of a software-defined network. In one example method, the VN controller 22 implements a data model-driven framework for initiating common operations on a set of network devices. These operations can be used by a user or network administrator to define and execute operations on devices in the data center 10 in a device-agnostic manner. In one example method, the framework is represented through a user interface (UI) 45 without hard-coding the part of the user or network administrator, and without the user or network administrator using CLI commands. In some such example methods, the framework employs a standard data model, an extensible UI, and automation tools (such as Ansible) in a data model-driven approach that utilizes the capabilities of the VN controller 22 to perform common operations on a selected set of network devices and display the results of the operations on the user interface 45. For example, this method can be used to enable sales engineering to automatically run selected commands, such as get commands, from the user interface (UI) 45 on data center devices. Other more complex commands can also be implemented based on the techniques described herein. In some example methods, device-agnostic commands are defined based on the job_template_type (i.e., device_operation) defined for each command.

[0113] These techniques eliminate the need for specific device operations / commands and remove the user interface dependency based on a specific device. Instead, common operations rely on input and output data models that can be dynamically represented. Additionally, in some example methods, the VN controller 22 uses role-based and profile-based device management to decouple the configuration / operation functions in each network device from specific common operations. By leveraging the concepts of role-based and profile-based device management, the framework can be used to automate a single operation across multiple device vendors, providing a consistent view of the results to the user regardless of the vendor-specific output / data provided by the specific vendor: device model.

[0114] In Figure 5 the example method, the UI 45 is displayed (200) within a graphical user interface indication associated with one or more network devices 12, 16, 18. The user selects (202) one or more of the network devices 12, 16, 18 via a marker, and then selects one or more device-agnostic commands to execute (204) on the selected network devices. Then, the VN controller 22 executes (206) the selected device-agnostic commands on the selected network devices.

[0115] In an example method, a graphical user interface includes generic device operation icons (e.g., buttons) that, when activated, initiate the execution of device-independent commands on one or more network devices from network 8. In one such example method, UI 45 displays a screen listing the generic device operation icons and a list of network devices configured for generic device operations. In some such example methods, a user or administrator selects one or more network devices on the network device list and initiates a generic device operation on the selected devices by clicking the generic device operation icon. In an example method, the execution of a single generic device operation command across each of the selected devices is considered a single job execution, even if the devices are of different series or different vendors.

[0116] Figures 6A to 6C Illustrated is a representative user interface for supporting device-independent commands in accordance with the techniques described herein. In Figures 6A to 6C the example method, each device-independent command is a generic device operation. For example, Figure 6A UI 45 in shows a generic device operation button 300 and a list 302 of network devices 12, 16, 18. In Figure 6A the example method, actuation of the generic device operation button 300 executes the device-independent command. In Figure 6A the example method. The network devices in list 302 can be selected by clicking on the box 304 to the left of the marker identifying each network device. In some example methods, other attributes associated with the network devices (such as IP address or vendor name) are listed along with the device markers. In some example methods, clicking button 300 causes the display of Figure 6B the screen shown in.

[0117] In Figure 6B the example method illustrated, UI 45 displays a list 306 of available generic device operations 308 and an icon 310 (e.g., a button) associated with the initiation of the selected device-independent command. In an example method, there are a number of device-independent commands, and each command can be run as a separate job execution. In one such example method, the available device-independent commands are displayed via a drop-down menu in UI 45. That is, list 306 is displayed as a drop-down menu and can be used to select one or more of the available device-independent commands.

[0118] In some example methods, a user selects one of the device-independent commands 308 (listed herein as generic device operations) from list 306 and initiates the execution of the operation by, for example, clicking icon 310. In some example methods, clicking icon 310 causes the display of Figure 6D the screen shown in.

[0119] In some example methods, when a user selects one of the device-independent commands 308 from the list 306, the selected command is highlighted, as Figure 6C shown for the "Display Configuration Interface" command, and Figure 6A a list 311 of the devices selected in Figure 6D appears in a window to the right of the list 306 of available general device operations 308. The user can then initiate the execution of the operation by, for example, clicking on the icon 310, which in turn can cause the display of a screen as

[0120] In Figure 6D the example method illustrated in Figure 6B the UI 45 displays one or more input parameters 312, 314 associated with the device-independent command selected in Figure 6D In the example shown in Figure 6A input parameter 312 is a parameter associated with an interface filter, while input parameter 314 is a parameter associated with an interface type. In some such example methods, the UI 45 also displays in a separate window

[0121] Figure 6E a list 311 of the devices selected in Figure 6D In some example methods, the user enters any desired parameters in a single parameter field (not shown) in the parameter name: parameter property pair. Figure 6E illustrates an example method where input parameter 314 is a drop-down menu for selecting between interface types, and where selecting "Add" located under parameter 312 in

[0122] In an example method, each device-independent command job is executed asynchronously in the background, and a prouter_uve object is returned for each device. In some example methods, when the device-independent command job is completed, a result page is displayed in the UI 45, which shows the command output for each selected network device. If there is an error when running the device-independent command on any of the selected network devices, the error is logged via a job log summary message. In some such example methods, the prouter_uve object for each selected network device carries the operation result of each device and is displayed in the result page within the UI 45.

[0123] Figures 6F to 6H illustrates Figure 6C a representative result page for the selected commands. In an example method, the result page 322 includes a window Figure 6A listing the devices 311 selected in Figure 6F to run the "Show Configured Interfaces" command. In an example method, the result page 322 is arranged as a series of tabbed windows: a "Results" window 324, a "Results-JSON" window 326, a "Job Status" window 328, and a "Job Status-JSON" window 330. As can be seen from Figure 6F in an example method, the tabbed "Results" window 324 displays the results 332 received for the devices 334 selected from the list 311. In an example method, the results of the commands on the selected devices 334 are displayed in the "Results" window 324, and the parameters displayed, as well as the order and format of the display, are defined by the output mode of the device-independent command.

[0124] Figure 6G illustrates an example method for displaying the results 332 received for the devices 334 selected from the list 311. In an example method, in Figure 6G the "Results-JSON" window 326, the results of the commands on the selected devices 334 are displayed, and the results of the commands are nested JSON files. In some such example methods, the parameters displayed, as well as the order and format of the display, are defined by the output mode of the device-independent command.

[0125] Figure 6H illustrates an example method for listing the job status 336 of the user-defined device-independent commands executed on the devices 304 shown in the list 302 selected from Figure 6A In an example method, in Figure 6HThe job status of commands executed on the selected device 334 is displayed in the "Job Status" window 328. Ideally, for any particular device selected, the information displayed in the "Job Status" window 328 is consistent, reflecting the successful execution of commands on each device. In one example method, the job log tracks the progress of the entire job (across all selected devices), and the job logs of all selected devices chosen from the list 302 shown in Figure 6A are displayed on the "Job Status" window 328. In one such example method, the window 328 also displays a final summary message.

[0126] The "Job Status - JSON" window 330 is not shown. In some example methods, the "Job Status - JSON" window 330 displays a JSON file presenting the information shown in Figure 6H .

[0127] In some example methods, the current number of devices that can be selected for device - independent commands is constrained to a maximum number of devices per job execution (e.g., 20). For example, this method may be necessary when the time to poll the selected devices for output causes the application to be clumsy when greater than the maximum number of devices. In one example method, the UI 45 polls all the prouter_uves of the current job and displays the output of each device across the representation of the devices. If some devices fail to execute, the job_log also carries an error message. In some example methods, the output of the failed devices is not shown.

[0128] In one example method, the VN controller 22 provides an mechanism through which users and administrators can add new device - independent commands, such as, for example, Figure 3B the command presentation unit 64. In one such example method, the user creates a new device - independent command type, defines the input and output modes of the new command type, identifies the devices in response to the new command type, and defines the actions to be taken during operation. Figure 7 is a flowchart illustrating an example method for creating a new device - independent command on a network device of a software - defined network according to the techniques described herein in Figure 3A and Figure 3B . In the example method of Figure 7 , the user creates a job template object type (350) for the new device - independent command. In one such example method, the user creates a new job_template for each new device - independent command and sets the job_template_name to the command name. In one example method, each new job template is stored in the job template data store 90 shown in Figure 3B .

[0129] In an example method, Ansible is used to define new device-agnostic commands. As noted above, Ansible is a provisioning, configuration, and deployment tool that relies on scripts to define sequences of tasks. Ansible uses scripts to describe automation jobs as tasks via a human-readable data serialization language called YAML. YAML is a standardized language commonly used for configuration files. However, it can be used in any application for storing data. YAML is very easy for humans to understand, read, and write, and can be advantageously used in applications such as Ansible to describe and document scripts.

[0130] In an example method, the VN controller 22 defines a common Ansible script (e.g., " / fabric_ansible_playbooks / operational_command.yml") referenced by each device-agnostic command job_template. In some Ansible-based example methods, the user adds the job_template object type to the predef_payloadsjson file (e.g., fabric_ansible_playbooks / conf / predef_payloads.json), sets the job_template_type to device_operation, thus marking the new template as a device-agnostic command template. In some example methods, changing the predef_payloads.json may require restarting the docker container associated with the device-agnostic command (e.g., config_api_1_xxxx docker). A JavaScript Object Notation (JSON) file is a file represented in JavaScript Object Notation, which is an open standard file format for exchanging data.

[0131] Figure 8 is a block diagram illustrating an example job template associated with a device-agnostic command according to the techniques described herein. As noted above, in some example methods, each new device-agnostic command requires a new job_template. In Figure 8 the example shown, the new device-agnostic command is titled "show_config_interfaces", and the new job_template is given the title "show_config_interfaces_template" at name 400. In Figure 8In the example shown, the job_template_type 402 is set to device_operation, which designates the job_template_type as a device-independent command template. Also, as noted above, all device-independent command job_templates reference the common Ansible script found at pointer 404 in this example, as follows:

[0132] / opt / contrail / fabric_ansible_playbooks / operational_command.yml

[0133] Return Figure 7 , in the example method of Figure 7 , the user defines the input and output modes (352) of the new device-independent command. For example, as Figure 8 shown. In some example methods, the form that can be used to capture the input parameters associated with the new device-independent command is defined. In one such example method, the form is based on the input mode defined for the device-independent command (e.g., input or input_ui) and is defined in the folder path:

[0134] opt / contrail / fabric_ansible_playbooks / schema / <command_name>_schema.json

[0135] where command_name is the job_template_name minus "_template". For example, for a job_template named "show_config_interfaces_template", the input mode can be defined in the path:

[0136] opt / contrail / fabric_ansible_playbooks / schema /

[0137] show_config_interfaces_schema.json The UI input mode can also be used to define details such as the placeholders used or the order of the input parameters of the command. The mode can be stored in the data store 92 together with the user-defined device-independent command.

[0138] As noted above, in one example method, "executing" causes the execution job API in the backend to start. In one such example method, the job runs in the background and, if needed, updates are shown in the job log on the UI 45. When the job is complete, the UI 45 reproduces the page that shows the output of running the command on all devices where the command execution was successful.

[0139] Return to Figure 7 , the user identifies the devices 12, 16, 18 that support the new device-independent command (354), assigns a "role" (356) to the new device-independent command, and adds a task to the device-independent command script that defines the tasks to be executed during the device-independent command (358).

[0140] As noted above, the VN controller 22 as described above uses role-based and profile-based device management to decouple the configuration / operation functions in each of the network devices 12, 16, 18 from specific common operations. Each device-independent command has a "role" (usually defined by its name), and the operation is performed on the profile of the target device defined by its profile (in some instances, defined by its device model). By leveraging the concepts of role-based and profile-based device management, the framework can be used to automate a single operation across multiple device vendors, providing the user with a consistent view of the results regardless of the vendor-specific output / data provided by a particular vendor: device model.

[0141] The following defines an example method for defining a new common device command. In one example method, all the changes described are made inside the config_api docker. Some changes may require restarting the docker. This example illustrates the steps for adding the new common device command "show_config_interfaces" referenced above in Figure 8 . In the following example, at least one device in the structure is a Juniper SRX device, the command name is "show_config_interfaces", the vendor associated with the device to be tested is "Juniper", and the device family is "junos-es".

[0142] As noted above in Figure 7 and Figure 8As pointed out in the discussion, the first step is to create the job template object type. In an example method, the user creates a job template named <command_name> template of type "device_operation". In this example, the job template is named "show_config_interfaces_template" and the type is "device_operation". In an example method, the new job_template object type is added to the list of job_template object types in predef_payloads.json. In an example method, marking the job_template_type as "device_operation" ensures that the new command is listed in UI 45. The Playbook_uri 406( Figure 8 as shown) can be used for the file path to add any new command. However, in some example methods, there is only one common entry point for any command operation. In some example methods, the remaining fields of show_config_interfaces_template are populated, as Figure 8 shown.

[0143] Next, the input and output modes are discussed. In Figure 8 an example method shown, the schema file is left blank. In an example method, the user creates the necessary schema file by the name <command_name>_schema.json under the directory name / opt / contrail / fabric_ansible_playbooks / schema / . In an example method, the schema file includes one or more of the following: input mode, output mode, and UI mode. In an example method, the input mode of the show_config_interfaces command includes two input parameters: interface filter and interface type, as follows:

[0144]

[0145]

[0146] In an example method, the UI mode provides the selected UI features. In one such example method, in the case of a large number of input parameters, the UI mode includes UI placeholders and sorting of form data. One such instance is as follows:

[0147]

[0148]

[0149] In one example method, the output mode varies according to a specific generic device command. In the example shown by the show_config_interfaces device-independent command, the user may wish to primarily export the operational status information of the interfaces. In some example methods, the output mode is actually under command_output, as the output mode is the output mode of the job. Thus, the formats of any other commands are mostly similar except for the command_output part in the output mode. One such example is shown below:

[0150]

[0151]

[0152]

[0153]

[0154] A Jinja template is a template created using the Jinja2 templating language.

[0155] In this example method, the user then creates an appropriate role for the command show_config_interfaces. This could be as simple as naming the command.

[0156] Figure 9 Illustrated is a representative network device on which device-independent commands can be executed according to the techniques described herein. In Figure 9 the example method, the device is a router. It is defined by name 450, vendor 452, device family 454, product name 456, and management IP address 458. The show_config_interfaces command can be configured to operate on the Figure 9 router as detailed below.

[0157] Figure 10 Illustrated is an example directory structure that can be used to organize device-independent commands according to the techniques described herein. In Figure 10 the example method, the assigned role (i.e., "show_config_interfaces") defines directory 500, which houses a role folder 502 (i.e., "cmd_show_config_interfaces") for each device-independent command. In Figure 10 the example shown, the Ansible script associated with the command is stored in task folder 504, and the template associated with the command is stored in template folder 506. In one example method, the top-level directory path of the role folder is:

[0158] / opt / contrail / fabric_ansible_playbooks).

[0159] In Figure 10 the method shown, the user creates a task folder 504 under the cmd_show_config_interfaces folder 502 and adds an Ansible script 508 for each device 12, 16, 18 on which the command will be executed. In one such example method, the Ansible script for a device family is labeled <device_vendor>_<device_family>.yml. However, if the parsing of command_resp( Figure 11 shown) operates correctly on all device families under a given vendor, the Ansible script for the given vendor is labeled <device_vendor>.yml. Templates 510 (e.g., for Juniper, Cisco, or Arista devices) are listed under the template folder 506. In Figure 10 the example shown, the template is defined as a Jinja template. As noted above, Jinja2 is a template language for Python that stores its templates as Jinja templates. Other template languages can also be used.

[0160] Figure 11 Illustrated is an example Ansible script for a Figure 9 router. In Figure 11 the example Ansible script 550, the command 552, when executed by the command processing unit 62, uses the Juniper Junos CLI command to perform a task associated with the device-independent command "show_config_interfaces" at the device_management_ip address of one or more Juniper devices. Executing the show_config_interfaces device-independent command at the management IP address returns the result of the Juniper Junos CLI command for that router device. The response is then stored in the "command_resp" register 554 and logged in the variable command_response 556 for logical access by the show_config_interfaces device-independent command. The Ansible scripts for Arista and Cisco devices follow a similar convention but are adapted to the associated device commands for Arista and Cisco devices respectively.

[0161] As noted above, the user defines the script 550 corresponding to the equipment supplier and equipment series. In Figure 11 the example shown, the Ansible script 550 may already be labeled juniper_junos-es.yml. However, since the device-independent commands work across all Juniper devices, it is labeled with juniper.yml.

[0162] It should be noted that the number of tasks in each script is a function of the underlying device. The method described above for the Juniper router performs a single task, receives the result in a register, and transfers the register content to the variable used by the show_config_interfaces device-independent command. Products from other suppliers may require several tasks to obtain a similar result. However, the advantage of the above method is immaterial. As long as the script and template can be defined for each device, the same device-independent command can be run on any number of devices.

[0163] Figure 12 An example template for specifying device-independent commands on a network device according to an aspect of the present disclosure is illustrated. The user creates the template 600. Each template 600 is associated with an equipment series and an equipment supplier. In Figure 12 the example shown, the template 600 is a Jinja template parsed by the Jinja2 engine. In an example method, the same naming convention is used for the template with respect to the script. In this case, the template 600 may be labeled "juniper_junos-es.j2". However, since the parsing of the command_resp variable of the Ansible script 500 from Figure 11 succeeds across all Juniper equipment series, the template 600 may be labeled juniper.j2, as shown in Figure 10 the figure. In an example method, the parsing output of the Jinja2 engine is recorded in a structure similar to the command_outputjson structure defined in the output mode described above.

[0164] In an example method, the output of each device-independent command is stored as a JSON file in a folder labeled with the command name. An example of such a folder is as follows:

[0165] / opt / contrail / fabric_ansible_playbooks / generic_device_operations / <command_name>

[0166] In an example method, the JSON output of the parsing of command executions on a device is stored as in a file named <device_management_ip>.json. In Figure 10 the case of a device, the command execution result of the "show_config_interfaces" device-independent command is stored as 10.175.72.108.json.

[0167] Figure 13 Illustrated is an output file obtained by the execution of device-independent commands on a selected network device according to one aspect of the present disclosure. In Figure 13 the example method, the results read from the command_resp variable for each device for the device-independent commands subject to Figure 8 are stored in a JSON file as nested data, where a separate JSON file is stored for each device. In Figure 13 is shown a representative output file 700. In some example methods, the file 700 has a format defined by an output mode defined by a job template for the device-independent commands, as pointed out above.

[0168] Described above are advantageous ways of executing device-independent commands across products of different product vendors while achieving similar results. Operations on a device family can be expressed as tasks and templates using Ansible scripts and Jinja2 templates, respectively. In an example method, a graphical user interface 45 on a VN controller 22 provides means for defining common operations and applying device-independent commands to devices based on representative task and template files. The result provides flexibility for device-agnostic testing across heterogeneous networks.

[0169] These techniques eliminate the need for device-specific operations / commands and remove user interface dependencies based on a particular device. Instead, common operations rely on input and output data models that can be dynamically reproduced. Additionally, in some example methods, the VN controller 22 uses role-based and profile-based device management to decouple the configuration / operation functions in each network device from specific common operations. By leveraging the concepts of role-based and profile-based device management, the framework can be used to automate a single operation across multiple device vendors, providing a consistent view of the results to the user regardless of the vendor-specific output / data provided by a particular vendor: device model.

[0170] The techniques described herein (including the techniques in any of the foregoing portions) may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of an electronic circuit may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0171] If implemented in hardware, the present disclosure may relate to an apparatus, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be at least partially implemented by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0172] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a computer data storage medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0173] In some examples, the computer-readable storage medium may include a non-transitory medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, the non-transitory storage medium may store data that changes over time (e.g., in RAM or a cache).

[0174] The code or instructions may be software and / or firmware executed by a processing circuit including one or more processors, such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term “processor” as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A virtual network controller, comprising: A memory; And One or more processors, the processors being connected to the memory, Wherein the memory includes instructions that, when executed by the one or more processors, cause the processors to: Receive data indicating an input that defines the device-independent command including an output mode of the device-independent command, wherein when the device-independent command is executed, one or more operations are performed on the supported network devices, and the one or more operations are defined by a job template associated with the device-independent command; For each network device series among a plurality of network device series, and based on different command sets associated with the corresponding network device series among the plurality of network device series, select one or more tasks that, when executed, perform the one or more operations of the device-independent command on network devices from the corresponding network device series, wherein the tasks are defined by a device-independent command script for the device-independent command, the device-independent command script specifies the tasks in the form of a data serialization language, and the device-independent command script is referenced by the job template; Cause a first network device associated with the network device series among the plurality of network device series to perform the operations of the device-independent command on the first network device, including executing the tasks selected for the network device series among the plurality of network device series; Store output variable data received from the first network device in response to the first network device executing the tasks selected for the network device series among the plurality of network device series; and Output a result page for display, the result page showing the command output for the first network device based on the output variable data, wherein the parameters displayed on the result page and the order and format of displaying the result page are defined by the output mode.

2. The virtual network controller according to claim 1, wherein the memory further includes instructions that, when executed by the one or more processors, cause the one or more processors to: Assign a role to each device-independent command.

3. The virtual network controller according to claim 1, wherein the memory further includes instructions that, when executed by the one or more processors, cause the one or more processors to: Store the tasks as device-independent command scripts associated with the device-independent command and associated with the device vendor associated with the network device.

4. The virtual network controller according to claim 1, wherein the memory further includes instructions that, when executed by the one or more processors, cause the one or more processors to: Store one or more modes associated with the device-independent command.

5. The virtual network controller according to claim 4, wherein the output mode defines how to present the information received from the first network device in response to the first network device executing the task selected for the network device series among the plurality of network device series.

6. The virtual network controller according to any one of claims 1 to 5, wherein the memory further includes instructions that, when executed by the one or more processors, cause the one or more processors to: output an indication of one or more locations of a graphical user interface for display, the one or more locations being configured to receive user input associated with the device-independent command during execution of the task selected for the network device series among the plurality of network device series, wherein the one or more locations include one or more of the following: a window configured to receive text input, and a drop-down menu.

7. A method, comprising: receiving, by a virtual network controller, data indicative of an input that defines the device-independent command including an output mode of the device-independent command, wherein when the device-independent command is executed, one or more operations are performed on a supported network device, the one or more operations being defined by a job template associated with the device-independent command; selecting, by the virtual network controller, for each network device series among the plurality of network device series and based on a different set of commands associated with the corresponding network device series among the plurality of network device series, one or more tasks that, when executed by a network device from the corresponding network device series, perform the one or more operations of the device-independent command, wherein the tasks are defined by a device-independent command script for the device-independent command, the device-independent command script being referenced by the job template; causing, by the virtual network controller, a first network device associated with the network device series among the plurality of network device series to perform the operations of the device-independent command on the first network device, wherein performing includes executing the task selected for the network device series among the plurality of network device series; storing, by the virtual network controller, output variable data received from the first network device in response to the first network device executing the task selected for the network device series among the plurality of network device series; and outputting, by the virtual network controller, a result page for display, the result page showing a command output for the first network device based on the output variable data, wherein the parameters displayed on the result page and the order and format of displaying the result page are defined by the output mode.

8. The method according to claim 7, wherein the method further includes assigning a role to each device-independent command.

9. The method according to claim 7, wherein the method further comprises storing the task as a device-independent command script associated with the device-independent command and associated with the device vendor of the network device.

10. The method according to any one of claims 7 to 9, wherein the method further comprises storing one or more modes associated with the device-independent command.

11. The method according to claim 10, wherein the mode comprises an input mode that defines input parameters associated with the device-independent command.

12. The method according to claim 11, wherein the output mode defines how to present information received from the first network device in response to the first network device executing the task selected for the network device series in the plurality of network device series.

13. The method according to any one of claims 7 to 9, further comprising: outputting an indication of one or more locations of a graphical user interface for display, the one or more locations being configured to receive user input associated with the device-independent command during execution of the task selected for the network device series in the plurality of network device series, wherein the one or more locations comprise one or more of the following: a window configured to receive text input, and a drop-down menu.

14. The method according to any one of claims 7 to 9, wherein the device-independent command script specifies the task in the form of a data serialization language.

15. A computer-readable storage medium comprising instructions that, when executed, cause one or more processors of a virtual network controller to: receive data indicative of an input that defines a device-independent command, wherein the device-independent command includes an output mode of the device-independent command that, when executed, performs one or more operations on a supported network device, the one or more operations being defined by a job template associated with the device-independent command; for each network device series in a plurality of network device series and based on different command sets associated with the corresponding network device series in the plurality of network device series, select one or more tasks that, when executed, perform the one or more operations of the device-independent command on network devices from the corresponding network device series, wherein the tasks are defined by a device-independent command script for the device-independent command, the device-independent command script specifies the task in the form of a data serialization language, and the device-independent command script is referenced by the job template; cause a first network device associated with the network device series in the plurality of network device series to perform the operations of the device-independent command on the first network device, wherein performing includes executing the task selected for the network device series in the plurality of network device series; Store the output variable data received from the first network device in response to the first network device executing the task selected for the network device series among the multiple network device series; and Output a result page for display, the result page showing the command output for the first network device based on the output variable data, wherein the parameters displayed on the result page and the order and format of displaying the result page are defined by the output mode.

16. The computer-readable storage medium according to claim 15, further comprising instructions that, when executed, cause the one or more processors to assign roles to each device-independent command.

17. The computer-readable storage medium according to claim 15, further comprising instructions that, when executed, cause the one or more processors to store the task as a device-independent command script associated with the device-independent command and associated with the device vendor associated with the network device.

18. The computer-readable storage medium according to claim 15, further comprising instructions that, when executed, cause the one or more processors to store one or more modes associated with the device-independent command, wherein the output mode defines how to present the information received from the first network device in response to the first network device executing the task selected for the network device series among the multiple network device series.

19. The computer-readable storage medium according to claim 18, wherein the output mode defines how to present the information received from the first network device in response to the first network device executing the task selected for the network device series among the multiple network device series.

20. The computer-readable storage medium according to any one of claims 15 to 19, further comprising instructions that, when executed, cause the one or more processors to output an indication of one or more positions of a graphical user interface for display, the one or more positions being configured to receive user input associated with the device-independent command during the execution of the task selected for the network device series among the multiple network device series, wherein the one or more positions include one or more of the following: a window configured to receive text input, and a drop-down menu.

Images