Packet loss monitoring in virtual routers

Abstract

Techniques are described for capturing a missing data packet and creating a modified missing data packet having missing information associated with the missing data packet to provide more detail of the missing data packet for further analysis and / or availability. For example, a computing device includes an internal communication channel, a process executing in user space, and a virtual router. The virtual router includes, for example, processing circuitry and a missing interface to the internal communication channel, where the virtual router is configured to: receive a data packet; in response to determining that the data packet is to be missing, create a modified missing data packet to include missing information associated with the data packet; and provide the modified missing data packet to the missing interface to transmit the modified missing data packet to the process via the internal communication channel.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 644,966, filed December 17, 2021, which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to computing networks, and more specifically, to monitoring for missing packets within a computer network. BACKGROUND

[0004] In a typical cloud data center environment, there are a large number of interconnected servers that provide compute and / or storage capacity to run various applications. For example, a data center can include a facility that hosts applications and services for users (i.e., customers of the data center).

[0005] For example, a data center can host all of the infrastructure equipment, e.g., network and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure spread across the globe with user support equipment located in various physical hosting facilities. In some examples, the infrastructure of a cloud data center can include a combination of physical devices, which can be referred to as “underlay resources,” that are linked to and communicate with various virtual resources, e.g., virtual servers, agents, and / or policy controllers, which can be referred to as “overlay resources.”

[0006] Various network devices included in a network fabric typically include mechanisms for locally or remotely configuring the devices, e.g., a management interface. Through interaction with the management interface of a network device, an administrator or other user can perform configuration tasks to configure the device, and the user can also execute operational commands on the device to manage, collect, and / or view operational data of the device. For example, a user can configure interface cards of the device, adjust parameters of supported network protocols, specify physical components within the device, modify routing information maintained by a router, access software modules and other resources resident on the device, and / or perform other configuration tasks. In addition, the user can also provide commands to view current operational parameters from the device, system logs, information related to network connections, network activity or other status information, and view and react to event information received from the device. SUMMARY

[0007] Generally, techniques are described for capturing a missing data packet and creating a modified missing data packet with missing information associated with the missing data packet to provide more details of the missing data packet for further analysis and / or usability. For example, a computing device (or referred to as a “compute node” or “server”) can provide a forwarding plane in the form of a virtual router that extends a network from physical routers and switches in a data center fabric into a virtual overlay network hosted in the computing device. The virtual router dynamically creates and manages one or more virtual networks that can be used for communication between application instances running on virtualized execution elements (e.g., virtual machines or containers). In accordance with the techniques described in this disclosure, the virtual router is configured to capture a missing data packet and create a modified missing data packet to include missing information associated with the missing data packet, which is then provided to an interface configured in the virtual router that communicates the modified missing data packet to a process, e.g., a packet analyzer tool.

[0008] In one example, the virtual router is configured to receive a data packet and, in response to determining that the data packet is to be missing (referred to herein as a “missing data packet”), the virtual router is configured to create a modified missing data packet to include missing information associated with the data packet. For example, the virtual router is configured to encapsulate the modified missing data packet with information that includes one or more attributes of the missing data packet (e.g., a missing type, when and where the missing occurred) and context-specific information associated with the missing data packet (e.g., routing information, table lookup information, etc.). In some examples, the virtual router is further configured with an interface to an internal communication channel of the computing device (referred to herein as a “missing interface”) to communicate the modified missing data packet to a process (e.g., a packet analyzer tool) via the internal communication channel to enable a user or administrator to further analyze the missing information associated with the data packet.

[0009] These techniques can provide one or more technical advantages. For example, the techniques described herein provide an administrator or user with more details associated with a missing data packet that occurred in a data plane without using logs that provide minimal information (e.g., a count of missing data packets). Moreover, the techniques described herein can enable monitoring of missing data packets during live networks without the need to recreate the missing data packets for debugging purposes, which can be difficult to recreate and cannot timely diagnose the cause of the missing data packet.

[0010] In another example, a method includes receiving, by a virtual router of a computing device, a data packet; responsive to determining that the data packet is to be dropped, creating a modified dropped data packet to include drop information associated with the data packet; and providing, by the virtual router, the modified dropped data packet to a drop interface of an internal communication channel of the virtual router to the computing device to transmit the modified dropped data packet to a process executing in a user space of the computing device via the internal communication channel.

[0011] In another example, a non-transitory computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a computing device to: receive a data packet; responsive to determining that the data packet is to be dropped, create a modified dropped data packet to include drop information associated with the data packet; and provide the modified dropped data packet to a drop interface of an internal communication channel of a virtual router to the computing device to transmit the modified dropped data packet to a process executing in a user space of the computing device.

[0012] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram illustrating an example computing infrastructure that can implement the techniques described herein;

[0014] Figure 2 is a block diagram illustrating an example data packet format of a modified dropped data packet that includes information specifying details about the dropped data packet, in accordance with the techniques described in this disclosure;

[0015] Figure 3 is a block diagram of an example computing device that includes a virtual router configured to capture a dropped data packet and encapsulate the dropped data packet with context-specific data to provide more details of the dropped data packet for further analysis and / or usability, in accordance with the techniques described in this disclosure;

[0016] Figure 4 is a flow diagram illustrating example operations of a virtual router to capture a dropped data packet and encapsulate the dropped data packet with context-specific data to provide more details of the dropped data packet for further analysis and / or usability, in accordance with the techniques described in this disclosure.

[0017] Throughout the specification and drawings like reference numerals refer to like elements. DETAILED DESCRIPTION

[0018] Figure 1is a block diagram illustrating an example computing infrastructure 8 in which examples of the technology described herein can be implemented. Generally, a data center 10 provides an operating environment for applications and services of customer sites 11 (illustrated as "customers 11") that have one or more customer networks coupled to the data center through a service provider network 7. For example, the data center 10 can host infrastructure equipment such as network and storage systems, redundant power, and environmental controls. The service provider network 7 is coupled to a public network 15, which can represent one or more networks managed by other providers and thus can form part of a large-scale public network infrastructure (e.g., the Internet). The public network 15 can represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a tier-3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider that operates the service provider network 7, an enterprise IP network, or some combination thereof.

[0019] Although the customer sites 11 and the public network 15 are primarily illustrated and described as edge networks of the service provider network 7, in some examples one or more customer sites 11 or the public network 15 can be a tenant network within the data center 10 or another data center. For example, the data center 10 can host multiple tenants (customers), each associated with one or more virtual private networks (VPNs), each of which can implement one of the customer sites 11.

[0020] The service provider network 7 provides packet-based connectivity to connected customer sites 11, data centers 10, and public networks 15. The service provider network 7 can represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 can implement, for example, multi-protocol label switching (MPLS) forwarding and in this case can be referred to as an MPLS network or MPLS backbone. In some cases, the service provider network 7 represents a plurality of interconnected autonomous systems, such as the Internet, that provides service from one or more service providers.

[0021] In some examples, the data center 10 can represent one of many geographically distributed network data centers. As Figure 1The data center 10 can be a facility that provides network services to customers, as shown by the example. Customers of the service provider can be collective entities, such as businesses, governments, or individuals. For example, a network data center can host network services for several businesses and end users. Other example services can include data storage, virtual private networks, traffic engineering, file services, data mining, scientific computing or supercomputing, etc. Although illustrated as a standalone edge network of the service provider network 7, elements of the data center 10, such as one or more physical network functions (PNFs) or virtualized network functions (VNFs), can be included within a kernel of the service provider network 7.

[0022] In this example, the data center 10 includes storage and / or computing devices (or “nodes”), such as servers 12A-12X (collectively, “servers 12”), interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, where the servers 12 are depicted as coupled to top-of-rack (TOR) switches 16A-16N. The servers 12 are computing devices and can also be referred to herein as “hosts” or “host devices.” Although only the server 12A coupled to the TOR switch 16A is shown in detail in Figure 1 In this example, the data center 10 includes storage and / or computing devices (or “nodes”), such as servers 12A-12X (collectively, “servers 12”), interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, where the servers 12 are depicted as coupled to top-of-rack (TOR) switches 16A-16N. The servers 12 are computing devices and can also be referred to herein as “hosts” or “host devices.” Although only the server 12A coupled to the TOR switch 16A is shown in detail in

[0023] The switching fabric 14 in the example shown includes interconnected top-of-rack (TOR) (or other “leaf”) switches 16A-16N (collectively, “TOR switches 16”) coupled to a distributed layer of chassis (or “spine” or “core”) switches 18A-18M (collectively, “chassis switches 18”). Although not shown, the data center 10 can also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular telephones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 can also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), evolved packet cores or other cellular network elements, and other PNFs.

[0024] In this example, TOR switch 16 and chassis switch 18 provide redundant (multi-homed) connectivity for server 12 to IP fabric 20 and service provider network 7. Chassis switch 18 aggregates traffic and provides connectivity between TOR switches 16. TOR switch 16 may be a network device providing Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching capabilities. Both TOR switch 16 and chassis switch 18 may include one or more processors and memory, and may execute one or more software processes. Chassis switch 18 is coupled to IP fabric 20, which can perform Layer 3 routing to route network traffic between data center 10 and customer site 11 via service provider network 7. The switching architecture of data center 10 is merely an example. Other switching architectures may have more or fewer switching layers, for example.

[0025] Server 12 may each represent a compute, server, or storage server. For example, each server 12 may represent a compute device, such as an x86 processor-based server configured to operate according to the techniques described herein. Server 12 may provide Network Functions Virtualization Infrastructure (NFVI) for an NFV architecture.

[0026] Any server in Server 12 can be configured with virtual execution elements using the resources of the virtualization server to provide isolation between one or more processes (applications) running on the server. "Hypervisor-based," "hardware-level," or "platform" virtualization refers to the creation of virtual machines, each containing a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualization / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is virtualized from the physical hardware of the host server, the executing application is isolated from the hardware of the host and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communication on the corresponding virtual network.

[0027] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in virtualized data centers (e.g., Data Center 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly permitted by security policies.

[0028] Virtual networks can use data center edge routers (10) Figure 1 (Not shown) Connects to and extends on top of Physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs). Virtual networks can also be used to implement Network Functions Virtualization (NFV) and service chaining.

[0029] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks, a physical underlay network consisting of IP fabric 20 and switching fabric 14, and a virtual overlay network. The role of the physical underlay network is to provide "IP fabric" which provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network can provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

[0030] As described further below with respect to virtual router 21 A, virtual routers running in virtualization servers 12 create a virtual overlay network on top of the physical underlay network using a dynamic "mesh" of tunnels between them. For example, these overlay tunnels can be MPLS tunnels over GRE / UDP or VXLAN tunnels, or NVGRE tunnels. The underlay physical routers and switches can not contain any per-tenant state of virtual machines or other virtual execution elements, e.g., any media access control (MAC) addresses, IP addresses, or policies. The forwarding tables of the underlay physical routers and switches can contain, for example, only the IP prefixes or MAC addresses of the physical servers 12. (Gateway routers or switches connecting virtual networks to the physical network are an exception and can contain tenant MAC or IP addresses.)

[0031] Virtual routers 21 of servers 12 typically contain per-tenant state. For example, a separate forwarding table (routing instance) can be contained for each virtual network. This forwarding table contains the IP prefixes (in the case of layer 3 overlays) or MAC addresses (in the case of layer 2 overlays) of virtual machines or other virtual execution elements (e.g., container pods) that are local to the server 12. No single virtual router 21 needs to contain all the IP prefixes or all the MAC addresses of all the virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that are locally present on the server 12 (i.e., that have at least one virtual execution element present on the server 12).

[0032] The control plane protocol between the network controller 24 or control plane nodes of physical gateway routers (or switches) can be BGP (and can be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3 VPNs and MPLS EVPN. The protocol between the network controller 24 and the virtual routers 21 can be based on XMPP, for example. The message schema exchanged over XMPP can be in accordance with Mackie et al., "BGP-Signaled End-System IP / VPNs," draft-ietf-l3vpn-end-system-06, December 15, 2016, which is incorporated by reference herein in its entirety.

[0033] In some examples, the servers 12 can implement "container-based" or "operating system" virtualization. "Container-based" or "operating system" virtualization refers to virtualization in which an operating system runs multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, e.g., containers provided by the open source DOCKER container application or CoreOS Rkt ("Rocket"). Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit a separate operating system and provide only an application suite and application-specific libraries. Typically, containers are executed by a host as isolated user space instances and can share an operating system and common libraries with other containers executed on the host. As a result, containers can require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communication over a corresponding virtual network.

[0034] In some examples, containers are managed by their host kernel to allow for limiting and prioritization of resources (CPU, memory, block I / O, network, etc.) without the need to launch any virtual machines, in some cases using namespace isolation features that allow for complete isolation of the application's (e.g., a given container's) view of the operating environment, including process tree, network, user identifiers, and mounted file systems. In some examples, containers can be deployed in accordance with Linux Containers (LXC), which is an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

[0035] The servers 12 host virtual network endpoints of one or more virtual networks that run on the physical network represented here by the IP fabric 20 and the switching fabric 14. Although primarily described in terms of a data center-based switching network, other physical networks such as the service provider network 7 can serve as the basis for one or more virtual networks.

[0036] Each server 12 can host one or more virtual execution elements, each having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a group of one or more containers (e.g., a pod), or another other virtual execution element, e.g., a Layer 3 endpoint of the virtual network. The term “virtual execution element” includes virtual machines, containers, and other virtualized computing resources that provide at least a partially independent execution environment for an application. The term “virtual execution element” can also encompass a pod of one or more containers. As Figure 1 As shown, the server 12A hosts virtual network endpoints in the form of virtual execution elements 22A-22N (collectively, “VEs 22”). The server 12 can execute as many virtual execution elements as possible given hardware resource limitations of the server 12. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use one virtual hardware component (e.g., an SR-IOV virtual function) enabled by the NIC 13A to perform packet I / O and receive / transmit packets on one or more communication links with the TOR switch 16A. Other examples of virtual network interfaces are described below.

[0037] Each server 12 includes at least one network interface card (NIC) 13, each of which includes at least one interface that exchanges data packets with TOR switches 16 over a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components for virtualized input / output (I / O). A virtual hardware component for I / O can be a virtualization of a physical NIC 13 (“physical function”). For example, in Single Root I / O Virtualization (SR-IOV) as described in the Peripheral Component Interface Special Interest Group SR-IOV Specification, a PCIe physical function of a network interface card (or “network adapter”) is virtualized to present one or more virtual network interfaces as “virtual functions” for use by individual endpoints executing on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resource, and a virtual function is an example of a virtual hardware component. As another example, one or more servers 12 can implement Virtio, a para-virtualization framework available for, e.g., Linux operating systems, that provides an emulated NIC function as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 can implement an open vSwitch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) for hosted virtual machines, where such vNICs can also represent a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. In some cases, a virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, a virtual hardware component is an SR-IOV virtual function. In some examples, any of servers 12 can implement a Linux bridge that emulates a hardware bridge and forwards data packets between virtual network interfaces of the server or between virtual network interfaces of the server and physical network interfaces of the server. For Docker implementations of containers hosted by a server, a Linux bridge or other operating system bridge executing on the server that exchanges data packets between containers can be referred to as a “Docker bridge.” As used herein, the term “virtual router” can include an open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other device and / or software located on a host device and performing switching, bridging, or routing of data packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12.

[0038] Any of the NICs 13 can include an internal device switch to exchange data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch can be a virtual Ethernet bridge (VEB) to switch between SR-IOV virtual functions and, accordingly, between endpoints configured to use the SR-IOV virtual functions, where each endpoint can include a guest operating system. The internal device switch can also be referred to as a NIC switch, and for SR-IOV implementations, as an SR-IOV NIC switch. The virtual hardware components associated with the NIC 13A can be associated with a Layer 2 destination address, which can be assigned by the NIC 13A or a software process responsible for configuring the NIC 13A. The physical hardware components (or “physical functions” for SR-IOV implementations) are also associated with a Layer 2 destination address.

[0039] To exchange data between the virtual hardware components associated with the NIC 13A, the internal device switch can perform Layer 2 forwarding to exchange or bridge Layer 2 packets between the virtual hardware components and the physical hardware components of the NIC 13A. Each virtual hardware component can be on a virtual local area network (VLAN) of a virtual network for a virtual network endpoint that uses the virtual hardware component for I / O.

[0040] The one or more servers 12 can each include a virtual router 21 that performs one or more routing instances for a respective virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance can be associated with a network forwarding table. Each routing instance can represent a virtual routing and forwarding instance (VRF) for an Internet Protocol Virtual Private Network (IP-VPN). For example, a packet received by the virtual router 21A (illustrated as “vRouter 21A”) of the server 12A from the underlying physical network fabric of the data center 10 (i.e., the IP fabric 20 and the switching fabric 14) can include an outer header to allow the physical network fabric to tunnel a payload or “inner packet” to a physical network address of a network interface card 13A of the server 12A that is executing the virtual router. The outer header can include not only the physical network address of the network interface card 13A of the server, but also a virtual network identifier, such as a VxLAN tag or a Multiprotocol Label Switching (MPLS) label that identifies a virtual network and a respective routing instance executed by the virtual router 21A. The inner packet includes an inner header with a destination network address that conforms to a virtual network addressing space of the virtual network identified by the virtual network identifier.

[0041] The virtual routers 21 terminate the virtual network overlay tunnels and determine the virtual network of a received data packet based on the tunnel encapsulation header of the data packet and forward the data packet to the appropriate destination virtual network endpoint of the data packet. For example, for the server 12A, for each data packet that is outbound from a virtual network endpoint (e.g., VE 22A) hosted by the server 12A, the virtual router 21A appends a tunnel encapsulation header that indicates the virtual network of the data packet to generate an encapsulated or “tunneled” data packet and the virtual router 21A outputs the encapsulated data packet to a physical destination computing device, e.g., another one of the servers 12, via the overlay tunnel of the virtual network. As used herein, the virtual routers 21 can perform the operations of a tunnel endpoint to encapsulate internal data packets initiated by a virtual network endpoint to generate tunneled data packets and to decapsulate tunneled data packets to obtain internal data packets for routing to other virtual network endpoints. In Figure 1 In examples, the virtual routers 21 can be implemented in the kernel, user space as SR-IOV virtual functions or smart NICs.

[0042] The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across the servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration platform that provides a container-centric infrastructure for automating the deployment, scaling, and operation of containers to provide a container-centric infrastructure. In the context of virtualized computing infrastructures, “orchestration” generally refers to the provisioning, scheduling, and management of host servers available to the orchestration platform to virtual execution elements and / or applications and services executing on such virtual execution elements. In particular, container orchestration allows for the coordination of containers and refers to, for example, the deployment, management, scaling, and configuration of containers of host servers by a container orchestration platform. Examples of orchestration platforms include Kubernetes, Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0043] The elements of the automation platform of the computing infrastructure 8 include at least the servers 12, the orchestrator 23, and the network controller 24. Virtual execution elements can be deployed to a virtualization environment using a cluster-based framework in which a cluster's master node manages deployment and operation of containers to one or more of the cluster's minion nodes. The terms "master node" and "minion node" as used herein encompass different orchestration platform terminology for similar devices, with the primary distinction being between a management element of a cluster and a virtual execution element host device of a cluster. For example, the Kubernetes platform uses the terms "master node" and "minion node," while the Docker Swarm platform refers to a swarm manager and swarm nodes.

[0044] The orchestrator 23 and the network controller 24 together implement the controller 5 of the computing infrastructure 8. The orchestrator 23 and the network controller 24 can be executed on separate computing devices, or on the same computing device. Each of the orchestrator 23 and the network controller 24 can be a distributed application executed on one or more computing devices. The orchestrator 23 and the network controller 24 can implement respective master nodes for one or more clusters, each cluster having one or more minion nodes implemented by respective servers 12. Generally, the network controller 24 controls network configuration of the data center 10 fabric, e.g., establishing one or more virtual networks for packet communication between virtual network endpoints. The network controller 24 provides a logically and in some cases physically centralized controller for facilitating operation of one or more virtual networks within the data center 10. In some examples, the network controller 24 can operate in response to configuration input received from the orchestrator 23 and / or an administrator / operator. Additional information regarding the network controller 24 in connection with other devices of the data center 10 or other software-defined network operations is found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled "PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS"; and U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled "Tunneled Packet Aggregation for Virtual Networks," each of which is incorporated by reference herein as if fully set forth herein. U.S. Patent Application No. 14 / 226,509 also includes further description of virtual routers, e.g., the virtual router 21 A.

[0045] Generally, orchestrator 23 controls the deployment, scaling, and operation of virtual execution elements in server 12 cluster, and provides a computing infrastructure, which can include a container-centric computing infrastructure. Orchestrator 23, and in some cases network controller 24, can implement a respective cluster master for one or more Kubernetes clusters. As one example, Kubernetes is a container management platform that provides portability across public and private clouds, which can both provide virtualized infrastructure for the container management platform.

[0046] According to the techniques described in this disclosure, a virtual router is configured to capture a missing packet and create a modified missing packet having missing information associated with the missing packet to provide more details of the missing packet for further analysis and / or availability. In some examples, the virtual router is configured to provide the modified missing packet to an interface for communication to a process executing in a user space of the virtual router for display and / or further analysis of the modified missing packet. Figure 1 In the illustrated example, an administrator or user can use controller 5 to configure virtual router 21 A to capture missing packets, create modified missing packets having missing information specifying details about the packets, and provide the modified missing packets to interface 28 (referred to herein as a “missing interface 28”) for communication of the modified missing packets to a process executing in a user space of virtual router 21 A for display and / or further analysis of the modified missing packets. For example, an administrator or user can use controller 5 to configure virtual router 21 A to capture missing packets and provide modified missing packets for all captured missing packets, missing packets based on a particular missing type, and / or missing packets based on a particular missing type and originating from a particular host (e.g., one or more VEs 22). Missing types can include fragment missing, interface missing, packet flow missing, network congestion missing, missing due to routing errors, etc. In some examples, an administrator or user can use controller 5 to further specify parameters to control when virtual router 21 A provides capture and monitoring of missing packets according to the disclosed techniques. For example, an administrator or user can use controller 5 to specify that virtual router 21 A provide capture and monitoring of missing packets after detecting a particular number of missing packets of a particular missing type.

[0047] The virtual router 21 A can also be configured to, in response to determining that the data packet is to be dropped, create a modified dropped data packet (e.g., by modifying the dropped data packet or a copy of the dropped data packet) to include information specifying details about the dropped data packet. As one example, the virtual router 21 A can encapsulate information specifying details about the dropped data packet, such as one or more attributes of the dropped data packet (e.g., what, where, and when the drop occurred) and context-specific information associated with the dropped data packet (e.g., why the data packet was dropped), into the modified dropped data packet. The one or more attributes of the dropped data packet can include information such as a type of drop, a location where the drop occurred (e.g., a location in software implementing the virtual router, an interface that received the dropped data packet, etc.), and / or a time when the drop occurred (e.g., a timestamp). The context-specific information associated with the dropped data packet can indicate a reason for determining that the data packet is to be dropped (e.g., routing information for the dropped data packet, an error caused by a lookup table, etc.). The context-specific information can vary depending on the type of drop. As further described below, the dropped data packet is specified in a payload of the modified dropped data packet, the one or more attributes of the dropped data packet are specified in a first portion of a header of the modified dropped data packet, and the context-specific information associated with the dropped data packet is specified in a second portion of the header of the modified dropped data packet.

[0048] The virtual router 21 A is configured to provide the modified dropped data packet to the drop interface 28 for communication of the modified dropped data packet to a process (or an external packet analyzer tool) executed by the virtual router 21 A via the internal communication channel. In some examples, the process is provided by the server 12A (e.g., a TCPDUMP process executed by a command line interface (CLI)) or is an external packet analyzer tool (e.g., Wireshark). In some examples, the virtual router 21 A can provide the modified dropped data packet to the controller 5 for additional availability (e.g., via a virtual router agent of the server 12A).

[0049] Once configured to capture and modify a missing data packet, the virtual router 21 A can receive a data packet from an interface (e.g., the virtual network interface 26A or the NIC 13A) and determine that the data packet will be missing (e.g., due to a fragment loss caused by the fragment table being full). In response, the virtual router 21 A creates a modified missing data packet (e.g., modifying the original data packet or a copy of the original data packet) and includes one or more attributes of the missing data packet and context-specific information associated with the missing data packet. For example, the virtual router 21 A can encapsulate a header of the modified missing data packet that includes a first portion of the header that specifies attributes of the missing data packet and a second portion of the header that specifies the context-specific information associated with the missing data packet. The missing interface 28 can obtain the modified missing data packet that includes the one or more attributes and the context-specific information associated with the missing data packet, which in turn can be communicated by the missing interface 28 to a process of the virtual router 21 A or an external packet analyzer tool from which an administrator or user can perform further analysis of the modified missing data packet.

[0050] Figure 2 FIG. 2 is a block diagram illustrating an example data packet format of a modified missing data packet that includes information specifying details about a missing data packet, in accordance with the techniques described in this disclosure. Figure 2 The modified missing data packet 200 in the example of FIG. 2 can represent a modified missing data packet generated by the virtual router 21 A described in FIG. 1. Figure 1 The modified missing data packet 200 in the example of FIG. 2 can represent a modified missing data packet generated by the virtual router 21 A described in FIG. 1.

[0051] The modified missing data packet 200 includes a header that includes a first portion that specifies one or more attributes 204 of the missing data packet. In this example, the attributes 204 can include a loss type 204A, a file name 204B, a line number 204C, a version 204D, a timestamp 204E, and / or an interface 204F. For example, the loss type 204A can specify a type of loss, e.g., a fragment loss, an interface loss, a data packet stream loss, a network congestion loss, a loss caused by a routing error, etc. The attributes 204 can also include a location where the loss occurred. For example, the server 12A can implement the virtual router 21 A with a software application. In this example, the attributes of the missing data packet can include a file name 204B of the software application that implements the virtual router 21 A, a line number 204C within the software application where the loss occurred, a version 204D of the software application, and / or an interface 204F (e.g., an interface name) that captured the missing data packet. In some examples, the attributes 204 of the modified missing data packet 200 can include a time of the missing data packet, e.g., the timestamp 204E. The attributes 204 of the modified missing data packet 200 are just one example and can include more or less information about the content, location, and time of the data packet loss.

[0052] The modified lost packet 200 includes a header that includes a second portion that specifies context-specific information 206 that describes why the lost packet was lost. The context-specific information 206 can vary depending on the type of lost packet. As one example, if the type of loss is a fragment loss, the modified lost packet 200 can include context-specific information about the fragment loss, such as a next hop identifier 206A of the lost packet, a flow identifier 206B associated with the lost packet, a route 206C of the lost packet (e.g., an address obtained from a route lookup), and / or a note 206D about the cause of the fragment loss (e.g., the fragment table is full). Other types of loss can include different context-specific information that can be used to determine the cause of the lost packet for that type of loss. Although the header that includes the loss information (e.g., the attributes 204 and the context-specific information 206) is described as part of the header, the loss information can be specified in multiple headers.

[0053] The modified lost packet 200 includes a payload 208 that specifies the lost packet (e.g., the original lost packet or a copy of the lost packet). In some examples, the modified lost packet 200 includes an Ethernet header 202 that specifies a placeholder Ethernet header to enable processes and / or packet analyzer tools to identify the modified lost packet. The format of the modified lost packet 200 is merely one example, and can alternatively be configured in any format supported by a particular packet analyzer tool.

[0054] Figure 3 is a block diagram of an example computing device that includes a virtual router configured to capture lost packets and create modified lost packets with loss information associated with the lost packets to provide more details of the lost packets for further analysis and / or usability in accordance with the techniques described in this disclosure. Figure 3 The computing device 300 of FIG. 3 can represent a real or virtual server and can represent Figure 1of any server 12 of the example instances. In this example, the computing device 300 includes a bus 342 that couples the hardware components of the computing device 300 hardware environment. The bus 342 couples a network interface card (NIC) 330, a storage disk 346, and one or more microprocessors 310 (hereinafter “microprocessor 310”). The NIC 330 can be SR-IOV capable. In some cases, a front side bus can couple the microprocessor 310 and the storage device 344. In some examples, the bus 342 can couple the storage device 344, the microprocessor 310, and the NIC 330. The bus 342 can represent a Peripheral Component Interconnect (PCI) Express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to the bus 342. In some examples, components coupled to the bus 342 control DMA transfers between components coupled to the bus 342.

[0055] The microprocessor 310 can include one or more processors each including independent execution units to execute instructions in conformance with an instruction set architecture, the instructions stored in a storage medium. The execution units can be implemented as separate integrated circuits (ICs) or can be combined within one or more multi-core processors (or “many core” processors), each multi-core processor implemented using a single IC (i.e., a chip multiprocessor).

[0056] The disk 346 represents a computer-readable storage medium including volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules or other data. The computer-readable storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), EEPROM, flash memory, CD-ROMs, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the microprocessor 310.

[0057] The main memory 344 includes one or more computer-readable storage media that can include random access memory (RAM) such as various forms of dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM, or static RAM (SRAM), flash, or any other form of fixed or removable storage medium that is accessible to the computer and which can be used to transfer or store information in the form of instructions or data structures, program code, and program data. The main memory 344 provides a physical address space consisting of addressable locations.

[0058] A network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange data packets using links of an underlying physical network. The interfaces 332 can include port interface cards having one or more network ports. The NIC 330 can also include on-card memory, e.g., for storing packet data. Direct memory access transfers between the NIC 330 and other devices coupled to the bus 342 can read / write from / to the NIC memory.

[0059] The memory 344, the NIC 330, the storage disks 346, and the processor 310 can provide an operating environment for a software stack including an operating system kernel 314 executing in kernel space. The kernel 314 can represent, for example, a Linux, a Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows Server operating system kernel available from Microsoft Corporation. In some instances, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-based Virtual Machine (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can encompass a virtual machine manager (VMM). The operating system including the kernel 314 provides an execution environment for one or more processes in user space 345.

[0060] The kernel 314 includes a physical driver 325 that uses the network interface card 330. The network interface card 330 can also implement SR-IOV to enable sharing of physical network functions (I / O) among one or more virtual execution elements, e.g., virtual execution elements 302A-302B (collectively, “virtual execution elements 302”), such as one or more virtual machines or containers. A shared virtual device, such as a virtual function, can provide dedicated resources so that each virtual execution element 302 can access dedicated resources of the NIC 330, thus the NIC 330 appears as a dedicated NIC to each virtual execution element 302. A virtual function can represent a lightweight PCIe function that shares physical resources with the physical function used by the physical driver 325 and other virtual functions. For an SR-IOV-capable NIC 330, the NIC 330 can have thousands of virtual functions available according to the SR-IOV standard, but for I / O intensive applications, the number of virtual functions configured is typically much smaller.

[0061] The computing device 300 can be connected to a physical network fabric that includes an overlay network that extends the fabric from physical switches to software or "virtual" routers coupled to physical servers of the fabric, including the virtual router 320. The virtual router can be a process or thread executed by a server 12 (e.g., Figure 1

[0062] In the example shown, the virtual router 320 can execute as a kernel module. In some examples, the virtual router can execute as a user space data plane development kit (DPDK) process (not shown). Other examples of a virtual router executing as a user space DPDK process are described in Indian Provisional Patent Application No. 202141008464, filed March 1, 2021, the entirety of which is incorporated by reference herein. Figure 3

[0063] The virtual router agent 316 connects to the network controller 24 using a channel that is used to download configuration and forwarding information. The virtual router agent 316 programs this forwarding state to a virtual router data (or "forwarding") plane represented by the virtual router 320. The virtual router agent 316 can execute as a user space process.

[0064] The virtual router 320 can replace and incorporate the virtual routing / bridging functionality of the Linux bridge / OVS module that is typically used for Kubernetes deployment pods. The virtual router 320 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 320 can perform network services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0065] ​​The virtual router 320 can be multithreaded and executes on one or more processor cores, e.g., forwarding cores 321A-321C (collectively, "forwarding cores 321"). The forwarding cores 321 of the virtual router 320 can implement a packet processing pipeline. The virtual router agent 316 can stitch the pipeline from the simplest to the most complex way depending on the operations to be applied to the packet. The virtual router 320 can maintain multiple instances of forwarding information. The virtual router 300 can use RCU (read-copy-update) locks to access and update tables. The virtual router 320 can include multiple queues, e.g., queues 334, mapped to the forwarding cores 321, which are used when processing packets within the processing pipeline.

[0066] To send packets to other compute nodes or switches, the virtual router 320 uses one or more physical interfaces 332. Typically, the virtual router 320 exchanges overlay packets with workloads provided by virtual execution elements, e.g., VMs or pods. The virtual router 320 has multiple virtual network interfaces, e.g., vifs. These interfaces can include a kernel interface vhost0 for exchanging packets with the host operating system; an interface pkt0 with the virtual router agent 316 for obtaining forwarding state from the network controller and sending exception packets. There can be one or more virtual network interfaces corresponding to one or more physical network interfaces 332.

[0067] Other virtual network interfaces of the virtual router 320 are used to exchange packets with workloads. Figure 3 The virtual network interfaces 312, 313 of the virtual router 320 are shown in FIG. 3. The virtual network interfaces 312, 313 can be any of the aforementioned types of virtual interfaces. In some cases, the virtual network interfaces 312, 313 are tap interfaces to enable a tap device, e.g., the virtual router 320, to forward packets to the user space 345.

[0068] In a kernel-based deployment of virtual router 320, virtual router 320 is installed as a kernel module within the operating system. Virtual router 320 records itself with the TCP / IP stack to receive packets from any desired operating system interface that it wants. The interface can be a bond, physical, tap (for VMs), veth (for containers), etc. Virtual router 320 in this mode relies on the operating system to send and receive packets for different interfaces. For example, the operating system can expose a tap interface supported by a vhost-net driver to communicate with a VM (e.g., virtual execution element 302). Once virtual router 320 records the packets from this tap interface, the TCP / IP stack sends all packets to it. Virtual router 320 sends packets via the operating system interface. In addition, NIC queues (physical or virtual) are handled by the operating system.

[0069] In a DPDK-based deployment of virtual router 320 (not shown), virtual router 320 is installed as a user space 345 application that links to DPDK libraries. This can result in faster performance than the kernel-based deployment, especially in the presence of high packet rates. DPDK Poll Mode Drivers (PMDs) use physical interfaces 332 instead of the kernel’s interrupt-based drivers. Registers of physical interfaces 332 can be exposed in user space 345 so that the PMDs can access them; physical interfaces 332 bound in this way are no longer managed by or visible to the host operating system, and the DPDK-based virtual router manages physical interfaces 332. This includes packet polling, packet processing, and packet forwarding. In other words, user packet processing steps are performed by the virtual router 220 DPDK data plane. When packet rates are high, the nature of this “polling mode” makes virtual router DPDK data plane packet processing / forwarding more efficient compared to the interrupt mode of the kernel-based deployment of virtual router 320. There are relatively fewer interrupts and context switches during packet I / O compared to the kernel mode virtual router, and in some cases interrupts and context switches during packet I / O can be avoided entirely.

[0070] The computing device 30 includes a virtual router agent 316 that controls the virtual network overlays of the computing device 300 and orchestrates the routing of packets within the computing device 300. Generally, the virtual router agent 316 communicates with the network controller 24 for the virtualization infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, e.g., the computing device 300, more specifically, the virtual router 320 and virtual network interfaces, e.g., the virtual network interfaces 312 and 313. By configuring the virtual router 320 based on information received from the network controller 24, the virtual router agent 316 can support configuring network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0071] In one example, network packets (e.g., Layer 3 (L3) IP packets or Layer 2 (L2) Ethernet packets) generated or consumed by the virtual execution elements 302A-302B within a virtual network domain can be encapsulated in another packet (e.g., another IP or Ethernet packet) that is transported by the physical network. Packets transported in the virtual network can be referred to herein as "internal packets," while the physical network packets can be referred to herein as "external packets" or "tunnel packets." The router 320 can perform encapsulation and / or decapsulation of virtual network packets within physical network packets. This functionality is referred to herein as tunneling, and can be used to create one or more overlay networks. Other example tunneling protocols that can be used in addition to IPinIP include IP over Generic Routing Encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), etc. The virtual router 320 performs tunneling encapsulation / decapsulation for packets originating / destined to any of the virtual execution elements 302, and the virtual router 320 exchanges packets with the virtual execution elements 302 via the bus 342 and / or bridges of the NICs 330.

[0072] As described above, the network controller 24 can provide a logically centralized controller to facilitate operation of one or more virtual networks. The network controller 24 may, for example, maintain a routing information base, e.g., one or more routing tables storing routing information for the physical network as well as one or more overlay networks. The virtual router 320 implements one or more virtual routing and forwarding instances (VRFs) 322A-322B for the respective virtual network for which the virtual router 320 operates as a respective tunnel endpoint. Generally, each VRF 322 stores forwarding information for a corresponding virtual network and identifies where a packet is to be forwarded and whether the packet is to be encapsulated in a tunnel protocol, e.g., with a tunnel header that can include one or more headers of different layers of the virtual network protocol stack. Each VRF 322 can include network forwarding tables storing routing and forwarding information for the virtual network.

[0073] The NIC 330 can receive a tunnel packet. The virtual router 320 processes the tunnel packet to determine the virtual network of the source and destination endpoints of the inner packet from the tunnel encapsulation header. The virtual router 320 can strip the Layer 2 header and the tunnel encapsulation header to forward the inner packet internally only. The tunnel encapsulation header can include a virtual network identifier, e.g., a VxLAN tag or an MPLS label, that indicates the virtual network, e.g., the virtual network corresponding to the VRF 322A. The VRF 322A can include forwarding information for the inner packet. For example, the VRF 322A can map the destination Layer 3 address of the inner packet to the virtual network interface 312. In response, the VRF 322A forwards the inner packet to the virtual execution element 302A via the virtual network interface 312.

[0074] The virtual execution elements 302a-302b can also source inner packets as destination virtual network endpoints. For example, the virtual execution element 302A can generate a Layer 3 inner packet destined for a destination virtual network endpoint executed by another computing device, i.e., not the computing device 300, or another virtual execution element, e.g., the virtual execution element 302B. The virtual execution element 302A sends the Layer 3 inner packet to the virtual router 320 via the virtual network interface 312 connected to the VRF 322A.

[0075] The virtual router 320 receives the inner packet and Layer 2 header and determines the virtual network of the inner packet. The virtual router 320 can use any of the above virtual network interface implementation techniques (e.g., macvlan, tap, veth, etc.) to determine the virtual network. The virtual router 320 uses the VRF 322A corresponding to the virtual network of the inner packet to generate an outer header for the inner packet, which includes an outer IP header for the overlay tunnel and a tunnel encapsulation header identifying the virtual network. The virtual router 320 encapsulates the inner packet with the outer header. The virtual router 320 can encapsulate the tunnel packet with a new Layer 2 header having a destination Layer 2 address associated with a device outside of the computing device 300 (e.g., a TOR switch 16 or one of the servers 12). If the destination is another virtual network endpoint executing on the computing device 300, the virtual router 320 routes the packet to the appropriate one of the virtual network interfaces 312, 313. The NIC 330 outputs the tunnel packet with the new Layer 2 header on an egress interface. If the destination is another virtual network endpoint executing on the computing device 300, the virtual router 320 routes the packet to the appropriate one of the virtual network interfaces 312, 313.

[0076] In some examples, a controller of the computing device 300 (e.g., the network controller 24 of FIG. 1) configures a default route in each of the virtual execution elements 302 to cause the virtual execution element 302 to use the virtual router 320 as the initial next hop for egress packets. In some examples, the NIC 330 is configured with one or more forwarding rules to cause all packets received from the virtual execution elements 302 to be switched to the virtual router 320. Figure 1

[0077] The network module 306 can obtain interface configuration data for configuring the virtual network interfaces for the virtual execution elements 302. The virtual router agent 316 operates as a virtual network control plane module for enabling the network controller 24 to configure the virtual router 320. The virtual network control plane (including the virtual router agent 316 for the worker nodes and the network controller 24) manages the configuration of the virtual networks implemented in the data plane in part by the virtual routers 320 of the worker nodes. The virtual router agent 316 communicates the interface configuration data for the virtual network interfaces to the network module 306 to enable the orchestration plane element (i.e., the network module 306) to configure the virtual network interfaces according to the configuration state determined by the network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Further, this can enable the network module 306 to obtain interface configuration data for multiple virtual network interfaces of the virtual execution elements and configure the multiple virtual network interfaces, which can reduce the communication and resource overhead inherent in invoking a separate network module 306 to configure each virtual network interface.

[0078] ​According to the techniques described in this disclosure, the virtual router 320 is configured to capture the missing packets and create modified missing packets with missing information associated with the missing packets to provide more details of the missing packets for further analysis and / or availability.

[0079] In Figure 3 In the example shown, the virtual router agent 316 can receive interface configuration data (e.g., via a command line interface (CLI) or other interface, or by receiving the virtual router agent 316 from a controller) to configure the missing interface 324 of the virtual router 320 to capture missing packets and create modified missing packets with missing information associated with the missing packets. In some examples, an administrator or user can configure the missing interface 324 to capture all missing packets. For example, the administrator or user can use the controller 5 to enter the command dropcap-i vif0 / 3 to cause the virtual router 320 to capture missing packets, create modified missing packets for all missing packets, and provide the modified missing packets to the missing interface 324 (vif0 / 3). In some examples, the administrator or user can further configure the missing interface 324 to capture missing packets of a particular missing type. For example, the administrator or user can use the controller 5 to enter the command dropcap-i vif0 / 3-t VP_DROP_FLOW_NO_MEMORY to cause the virtual router 320 to capture missing packets, create modified missing packets for packets of a flow that are missing due to no memory (VP_DROP_FLOW_NO_MEMORY), and provide the modified missing packets to the missing interface 324. In some examples, the administrator or user can further configure the missing interface 324 to capture missing packets of a particular missing type and / or originating from a particular host. For example, the administrator or user can use the controller 5 to enter the command dropcap-i vif0 / 3-t VP_DROP_FLOW_NO_MEMORY host 10.10.10.1 to cause the virtual router 320 to capture missing packets, create modified missing packets for packets of a flow that are missing due to no memory (VP_DROP_FLOW_NO_MEMORY), and provide the modified missing packets to the missing interface 324. In some examples, the administrator or user can configure the missing interface 324 to capture missing packets of only a particular missing type, only missing packets originating from a particular host device, missing packets of a particular missing type originating from a particular host, or any combination of missing parameters.

[0080] The computing device 300 can include an internal communication channel 366, e.g., a Linux socket, to enable the loss interface 324 to communicate the modified loss packets to one or more processes, e.g., the process 308 (e.g., TCPDUMP), executing within the user space 345 or outside of the computing device 300, e.g., a packet analyzer tool, via the internal communication channel 366.

[0081] Once the loss interface 324 is configured, any forwarding core 321, e.g., the forwarding core 321 A, can receive a packet and process the packet within its packet processing pipeline. The forwarding core can capture the packets processed within the packet processing pipeline that are determined to be lost and provide the modified loss packets to the loss interface 324 (e.g., all loss packets from a particular originating host by loss type) in accordance with the interface configuration data used to configure the loss interface 324. For example, the virtual router 320 can receive a packet from an interface (e.g., a NIC 330 or one of the virtual network interfaces 312, 313) and the forwarding core 321 A processes the packet in accordance with the packet processing pipeline. The forwarding core 321 A can determine that the packet will be lost at any point in the packet processing pipeline, e.g., due to a packet processing error (e.g., fragment loss, interface loss, packet stream loss, network congestion loss, loss due to lookup table errors, etc.) occurring within the packet processing pipeline.

[0082] In response to determining that the packet will be lost, the forwarding core 321 A can not lose the packet, but rather create a modified loss packet having one or more attributes (e.g., the attributes 204) of the lost packet and context-specific information (e.g., the context-specific information 206) of the lost packet. Figure 2 Figure 2 ​the modified loss packet, including a first portion of the header that specifies attributes such as a loss type, a file name of a software application that implements the virtual router 320, a line number in the source application where the loss occurred, a version of the software application, a timestamp of a time when the loss occurred, and / or an interface name. The header of the modified loss packet also includes a second portion that specifies context-specific information associated with the loss packet that indicates why the packet was lost (e.g., information that describes a cause of a packet processing error that occurred within the packet processing pipeline, etc.). The context-specific information encapsulated by the forwarding core 321 A can vary depending on the type of loss packet. As one example, the forwarding core 321 A of the virtual router 320 can receive a packet from the virtual network interface 312 and determine that the packet is to be lost due to fragmentation loss (which will be captured by the loss interface 324). In this example, the forwarding core 321 A can create the modified loss packet that includes one or more attributes that specify at least one virtual network interface attribute that indicates a virtual network interface (e.g., vif) on which the packet was received and other attributes such as a loss type (fragmentation loss), a file name of a software application that implements the virtual router 320, a line number within the source application where the loss occurred, a version of the software application, and a timestamp of a time when the loss occurred. The forwarding core 321 A can also include context-specific information in the modified loss packet to specify information such as a next hop identifier, a flow identifier, a route, and / or an annotation that indicates that the loss was due to the fragmentation table being full.

[0083] The forwarding core 321 A can queue the modified missing packet into at least one queue 334 of the virtual router 320, which is a queue associated with the missing interface 324. One forwarding core 321 (e.g., the forwarding core 321C) can operate as a service core for sending and receiving packets from different interfaces of the virtual router 320, can use an interrupt-based driver in the case of a kernel-based virtual router 320, or a tap driver in the case of a virtual router 320 installed as a user space 345 application, to send the modified missing packet in the queue 334 to the missing interface 324, thereby communicating the modified missing packet to a process 308 executing within the user space 345 or a process executed by a device external to the computing device 300 via the internal communication channel 366. Other forwarding cores 321 can similarly capture missing packets, create corresponding modified missing packets, and queue the modified missing packets into at least one queue 334 of the virtual router 320 from which a service core can send the modified missing packets to the missing interface 324 for communication of the modified missing packets to one or more processes.

[0084] In some examples, an administrator or user can use a command line interface (CLI) of the computing device 300 to display the modified missing packets for further analysis. For example, an administrator or user can use a process 308, e.g., a TCPDUMP process executed by the CLI, to obtain the modified missing packets from the missing interface 324 and display one or more attributes and context-specific information of the modified missing packets via a display. In some examples, an administrator or user can use a process executed by an external device, e.g., a packet analyzer tool (e.g., Wireshark) external to the computing device 300, which can obtain the modified missing packets from the missing interface 324 and display the modified missing packets. In some examples, the process 308 can represent a process executing within the user space 345 that causes the virtual router agent 316 to obtain the modified missing packets from the missing interface 324 and provide the modified missing packets to a controller (e.g., the controller 5 of Figure 1 for additional availability and / or processing.

[0085] Figure 4 is a flowchart illustrating example operations of a virtual router in accordance with the techniques described in this disclosure for capturing missing packets and modifying the missing packets with context-specific information to provide more details of the missing packets for further analysis and / or availability. For purposes of example, the operations are described with reference to the server 12A of Figure 1 and components of the server 12A, e.g., the computing device 300 of Figure 3 are described.

[0086] exist Figure 4 In one example, computing device 300 includes a virtual router 320 configured with a lost interface 324 to an internal communication channel 336 to deliver a modified lost data packet to process 308. In some examples, virtual router 320 may receive a data packet (402) and determine that the packet will be lost. For example, virtual router 320 may receive data packets from an interface (e.g., NIC 330 or one of virtual network interfaces 312, 313), and forwarding core 321A processes the data packets according to a packet processing pipeline. Virtual router 320 may determine that a data packet will be lost at any point in the packet processing pipeline, for example, due to packet processing errors occurring within the packet processing pipeline (e.g., fragment loss, interface loss, packet flow loss, network congestion loss, loss due to lookup table errors, etc.).

[0087] In response to determining that the packet will be lost, virtual router 320 can create a modified lost packet with loss information associated with the packet (e.g., by modifying the packet or a copy of the packet) instead of losing the packet (404). For example, forwarding core 321A can create a modified lost packet by including the modified lost packet, which has the lost packet and context-specific information associated with the lost packet (e.g., ...). Figure 2 One or more attributes of the context-specific information 206 (e.g., Figure 2 (Attribute 204). As an example, the virtual router 320 may specify the lost packet (original or copy of the lost packet) in the payload of the modified lost packet, and encapsulate the modified lost packet header with a first part of a header that specifies one or more attributes and a second part of a header that specifies context-specific information associated with the lost packet.

[0088] The virtual router 320 provides the modified missing data packet to a missing interface 324 to an internal communication channel to communicate the modified missing data packet to the process 308 via the internal communication channel 406. For example, the forwarding core 321 A can queue the modified missing data packet into at least one queue 334 of the virtual router 320 that is associated with the missing interface 324. Other forwarding cores (e.g., forwarding core 321 B) can also provide respective modified missing data packets to the missing interface 324 to communicate the respective modified missing data packets to the process 308 via the internal communication channel. One of the forwarding cores 321 (e.g., forwarding core 321 C) that can operate as a service core for sending and receiving data packets from different interfaces of the virtual router 320 can use an interrupt-based driver in the case of a kernel-based virtual router 320 or a tap driver in the case of a virtual router 320 installed as a user space 345 application to send the modified missing data packet in the queue 334 to the missing interface 324 to communicate the modified missing data packet to the process 308 (e.g., TCPDUMP of a CLI provided by the computing device 300 or an external packet analyzer tool) that an administrator or user can use to display the modified missing data packet for further analysis via the internal communication channel 366.

[0089] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units or components can be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of electronic circuitry can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chip sets.

[0090] If implemented in hardware, the present disclosure can involve the use of one or more appropriate devices, such as a processor or an integrated circuit device, such as an integrated circuit chip or chip set. Alternatively or additionally, if implemented in software or firmware, the techniques can at least partly be realized with a computer program product comprising a computer readable data storage medium having program code stored thereon, the program code being designed to, when executed, implement one or more of the above-described methods. For example, the computer readable data storage medium can store such program code that is executed by a processor.

[0091] The computer readable medium can form part of a computer program product, which can include packaging material. The computer readable medium can include a computer data storage medium, such as a random access memory (RAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and / or the like. In some examples, the article of manufacture can include one or more computer readable storage media.

[0092] In some examples, computer-readable storage media can include non-transitory media. The term "non-transitory" can indicate that the storage medium is not among the transient signals that can change with time. In certain examples, a non-transitory storage medium can store data that can, over time, change (e.g., in RAM or cache).

[0093] Code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within dedicated software modules or hardware modules configured for implementing this functionality.

Claims

1. A computing device comprising: an internal communication channel; a process executing in user space; a virtual router comprising: processing circuitry; a loss interface to the internal communication channel, wherein the virtual router is configured to: receive a data packet; in response to determining that the data packet is to be lost, create a modified loss packet to include loss information associated with the data packet by: encapsulating the loss information as a header of the modified loss packet, wherein the loss information comprises at least context-specific information associated with a reason for determining that the data packet is to be lost; and designating the data packet as a payload of the modified loss packet; and provide the modified loss packet to the loss interface for communicating the modified loss packet to the process via the internal communication channel.

2. The computing device of claim 1, wherein, the loss information comprises: one or more attributes associated with the data packet to be lost.

3. The computing device of claim 2, wherein, the one or more attributes comprise one or more of a loss type, a file name of a software application implementing the virtual router, a line number indicating an instruction executing at a time of occurrence of the determination to lose the data packet, a version of the software application, or a timestamp, or a virtual network interface of the virtual router.

4. The computing device of claim 3, wherein, the one or more attributes of the virtual network interface comprising the virtual router comprise: a virtual network identifier in the modified loss packet indicating the virtual network identifier on which the data packet was received.

5. The computing device of claim 2, wherein, the context-specific information comprises information describing a cause of a data packet processing error occurring within a data packet processing pipeline of the virtual router.

6. The computing device of claim 1, wherein, the virtual router executes as a kernel module of the computing device.

7. The computing device of claim 1, wherein, the virtual router executes as a user space data plane development kit (DPDK) process.

8. The computing device of claim 1, wherein, to create the modified loss packet, the virtual router is configured to: generate a copy of the data packet; and modify the copy of the data packet to include loss information associated with the data packet.

9. The computing device of claim 1, wherein, to create the modified loss packet, the virtual router is configured to: modify the data packet to include loss information associated with the data packet.

10. The computing device of claim 1, wherein, creating the modified loss packet is based on a particular loss type.

11. The computing device of claim 1, wherein, creating the modified loss packet is based on a data packet originating from a particular host.

12. The computing device of claim 1, wherein, creating the modified loss packet is based on a particular loss type and a data packet originating from a particular host.

13. The computing device of any of claims 1 to 12, wherein, to provide the modified loss packet to the loss interface, the virtual router is configured to: send, by a forwarding core of a plurality of logical cores of the virtual router, the modified loss packet to a queue associated with the loss interface; obtain, by a servicing core of the plurality of logical cores, the modified loss packet from the queue; and send, by the servicing core, the modified loss packet to the loss interface.

14. The computing device of claim 6, the queue associated with the loss interface comprises modified loss packets from other forwarding cores of a plurality of logical cores, and wherein the queue associated with the loss interface comprises modified loss packets from other forwarding cores of a plurality of logical cores, and wherein the virtual router is further configured to: obtain, by a service core from the queue, the modified dropped packet from other forwarding cores of the plurality of logical cores; and send, by the service core to the drop interface, the modified dropped packet from other forwarding cores of the plurality of logical cores.

15. A computer network packet processing method, comprising: receiving, by a virtual router of a computing device, a data packet; in response to determining that the data packet is to be dropped, creating a modified dropped packet to include drop information associated with the data packet by: encapsulating the drop information as a header of the modified dropped packet, wherein the drop information includes at least context-specific information associated with a reason for determining that the data packet is to be dropped; and designating the data packet as a payload of the modified dropped packet; and providing, by the virtual router, the modified dropped packet to a drop interface of an internal communication channel of the virtual router to the computing device to transmit the modified dropped packet to a process executing in a user space of the computing device via the internal communication channel.

16. The method of claim 15, wherein, encapsulating the drop information as the header of the modified dropped packet includes: encapsulating a first portion of the header to include one or more attributes associated with a data packet to be dropped; and encapsulating a second portion of the header to include context-specific information associated with the data packet to be dropped.

17. The method of any one of claims 15-16, wherein, providing the modified dropped packet to the drop interface of the virtual router includes: sending, by a forwarding core of a plurality of logical cores of the virtual router, the modified dropped packet to a queue associated with the drop interface; obtaining, by a service core of the plurality of logical cores, the modified dropped packet from the queue; and sending, by the service core, the modified dropped packet to the drop interface.

18. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method of any one of claims 15 to 17.

Images