Adaptive Address Translation Cache

Abstract

Systems and methods are used to optimize utilization of an address translation cache (ATC). A network interface controller (NIC) may write information that reserves one or more cache lines in a first level of the ATC to a second level of the ATC. The NIC may receive a request for direct memory access (DMA) of an untranslated address in a memory of a host computing system. The NIC may determine that the untranslated address is not cached in the first level of the ATC. The NIC may use the request and the second level of the ATC to identify a selected cache line in the first level of the ATC for eviction. The NIC may receive a translated address of the untranslated address. The NIC may cache the untranslated address in a selected cache line. The NIC may perform DMA using the translated address.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. non-provisional patent application Ser. No. 16 / 290,101, filed on Mar. 1, 2019, and entitled “ADAPTIVE ADDRESS TRANSLATION CACHES,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] The subject matter of the present disclosure relates generally to the field of computer memory and, more particularly, to systems and methods for optimizing utilization of address translation caches. Background Art

[0004] A computing system may include an input / output (I / O) memory management unit (IOMMU) to provide address translation for direct memory access (DMA) from an I / O device (e.g., a network adapter, a graphics processing unit (GPU), a storage controller, etc.) to the main memory of the computing system, and to protect the memory from problematic devices by limiting the area of ​​main memory that the I / O device can access. The IOMMU can intercept DMA transactions and utilize the I / O page table to determine whether access is allowed and resolve the actual physical address in the main memory to be accessed. To further improve performance, the IOMMU may include an I / O translation lookaside buffer (IOTLB), which may operate as a cache to speed up address resolution. However, due to the additional overhead required for the walkthrough of the I / O page table, an IOTLB cache miss may represent a significant performance degradation (relative to an untranslated addressing scheme).

[0005] To mitigate IOTLB cache misses, the I / O devices themselves may include a device-side IOTLB (sometimes called a device translation lookaside buffer (TLB) or address translation cache (ATC)) and support the Address Translation Service (ATS), which was developed by the Peripheral Component Interconnect Special Interest Group. Peripheral Component Interconnect Express The ATS is specified in the base specification. By implementing ATS in the I / O device, ATC can offload at least some address resolution processing from the central processing unit (CPU) and / or IOMMU and increase the aggregate size of the IOTLB (e.g., IOMMU IOTLB plus ATC). However, current implementations of ATS have various disadvantages. For example, a computing system is expected to execute more than a thousand applications and system images (e.g., virtual machines, containers, or other virtual partitions of a physical server) at a time, but current implementations of ATS may not be able to support these loads. In addition, ATS is a general specification designed to provide interoperability for all types of PCIe endpoints, and may not be optimal for specific types of workloads, heterogeneous workloads, and certain types of I / O devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a more complete understanding of the present disclosure and its features and advantages, reference is made to the following description in conjunction with the accompanying drawings, in which:

[0007] Figure 1 A block diagram illustrating an example of a network according to an embodiment;

[0008] Figure 2A-2B A block diagram illustrating an example of a computing system according to some embodiments;

[0009] Figure 3A A block diagram illustrating an example of a computing system for implementing I / O virtualization (IOV) and address translation services (ATS) according to an embodiment;

[0010] Figure 3B A block diagram showing an example of a communication protocol for an ATS according to an embodiment;

[0011] Figure 4 A block diagram illustrating an example of a network interface controller according to an embodiment; and

[0012] Figure 5 A sequence diagram illustrating an example process for optimizing utilization of an address translation cache according to an embodiment is shown. DETAILED DESCRIPTION

[0013] The detailed description set forth below is intended to be a description of various configurations of embodiments, and is not intended to represent a unique configuration in which the subject matter of the present disclosure can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for providing a more thorough understanding of the subject matter of the present disclosure. However, it is clear and obvious that the subject matter of the present disclosure is not limited to the specific details set forth herein, and can be practiced without these details. In some instances, structures and components are shown in block diagram form to avoid confusing the concept of the subject matter of the present disclosure.

[0014] Overview

[0015] Various aspects of the invention are set out in the independent claims, while preferred features are set out in the dependent claims. Features of one aspect may apply to each aspect alone or in combination with the other aspects.

[0016] Systems and methods are used to optimize utilization of an address translation cache (ATC). A network interface controller (NIC) may write information to reserve one or more cache lines in a first level of the ATC to a second level of the ATC. The NIC may receive a request for direct memory access to an untranslated address in a memory of a host computing system. The NIC may determine that the untranslated address is not cached in the first level of the ATC. The NIC may use the request and the second level of the ATC to identify a selected cache line in the first level of the ATC. The NIC may receive a translated address of the untranslated address. The NIC may cache the untranslated address in a selected cache line. The NIC may perform DMA using the translated address.

[0017] Example Embodiments

[0018] Virtualization can include various mechanisms for generating multiple simulated instances from a single physical instance of computing resources (e.g., processing, memory, I / O, storage, networking, etc.). Virtualization can improve resource utilization (e.g., maximum server capacity, minimum hardware cost, reduced energy use, etc.), accelerate and simplify network and operations management (e.g., system provisioning and deployment, system management, disaster recovery, and improve business agility (e.g., live migration of local computing resources, simplified migration from local computing to cloud computing and between different cloud providers, avoiding vendor lock-in, etc.). However, virtualization adds an additional software layer between the virtual instance and the hardware, which may have a negative impact on performance. For example, input / output (I / O) may become a bottleneck in a virtualized system due to the additional overhead of the virtualization layer (e.g., hypervisor, container engine, or other virtualization software).

[0019] Certain approaches can be taken to improve I / O performance in virtualized systems. For example, virtualized systems can incorporate IOMMUs to safely provide direct memory access (DMA) to I / O devices. However, IOMMUs increase memory access latency because address translation is required to protect memory from problematic devices and to implement virtual memory and other virtualizations. I / O translation lookaside buffers (IOTLBs) can reduce address resolution latency to some extent, but IOTLB cache misses can significantly increase memory access latency due to the traversal of the I / O page tables to retrieve the target physical address. In addition, the IOTLB is a limited, scarce resource, and because the IOTLB is platform-dependent, over-reliance on the IOTLB can lead to vendor lock-in.

[0020] Another technology for improving I / O performance in virtualized systems is I / O virtualization, such as single root I / O virtualization (SR-IOV). SR-IOV can allow multiple system images (e.g., virtual machines, containers, or other virtual partitions of a physical server) to share physical I / O devices and enable system images to interact more directly with physical I / O devices. Therefore, the additional overhead of the virtualization layer (e.g., hypervisor, container engine, or other virtualization software) can be significantly reduced by pre-intercepting and processing each I / O operation by the virtualization layer. Although SR-IOV can provide direct delivery of I / O tasks, the delays caused by IOTLB cache misses and page traversals may not be suitable for ultra-large-scale workloads (e.g., executing thousands of system images simultaneously in a single computing system). In addition, even when a computing system runs a single system image, if this system image executes thousands of applications, a small number of applications with I / O intensive workloads, and / or one or more applications that access several different memory areas, IOTLB cache thrashing may still occur.

[0021] In addition to limiting the scale of current approaches to I / O in virtualized systems, current approaches may also suffer from inflexibility to a variety of workloads. For example, an IOTLB cache miss may depend on the page size used by the application. Different applications may utilize pages of different sizes. High performance computing (HPC) applications may have different quality of service (QoS) requirements than other applications. Different applications may also have different memory access patterns (e.g., variations in duration, rate, burstiness, etc.). Current systems are unable to accommodate these heterogeneous workloads (e.g., different numbers of system images, different numbers of applications, applications with different sized I / O workloads, applications with different priorities, applications with different I / O patterns, and other differences).

[0022] A strategy used to reduce IOTLB cache thrashing is to incorporate ATC in I / O devices and configure I / O devices to implement ATS, such as PCI In the base specification As described in the specification, which is incorporated herein by reference. ATC can alleviate address resolution delays to a certain extent, but ATC may not be suitable for large-scale heterogeneous workloads. Like the CPU's TLB, IOMMU's IOTLB, or other caches, the size of ATC may be limited because it can be a relatively expensive resource. Current cache replacement policies (e.g., least recently used (LRU)) for evicting existing entries from ATC can be "universal (one size fits all)" and static. Depending on the nature of the (one or more) I / O workloads, this inadaptability can degrade the performance of the virtualized system because each eviction requires the release of a new page and causes IOTLB thrashing. This is particularly evident in containerized computing systems, where there may be more than a thousand containers that need to pass operations. Although the computing system may have the ability to support such ultra-large-scale workloads, the IOTLB (e.g., IOMMU IOTLB and ATC) may become a bottleneck.

[0023] Various embodiments of the present disclosure can overcome these and other deficiencies of the prior art by utilizing an ATC that can intelligently adapt to heterogeneous workloads and applying one or more ATC entry replacement policies that can rely on multiple different criteria, such as user preferences, rate limits, quality of service (QoS), and workload type, as well as other information that identifies or distinguishes a group of network traffic.

[0024] In some embodiments, a user may partition the number or percentage of entries for the ATC based on preferences. For example, a user may request the NIC (via a network management system or other software) to set aside 75% of the ATC for virtual NIC (VNIC) identifiers and process address space identifiers (PASIDs) associated with HPC applications. In response, the NIC may reserve a specified number or percentage of entries for the specified VNIC identifiers and PASIDs. Other user preferences that may dictate ATC entry replacement policies may depend on the capabilities of the NIC, such as whether the NIC includes hardware support for traffic classification, packet matching, transmission control protocol (TCP) offload, and the like. If the NIC is capable of providing these features, the user may define more refined ATC entry replacement policies based on traffic class, flow matching, packet matching, deep packet inspection, and other information that identifies or distinguishes network traffic.

[0025] In some embodiments, the NIC may receive QoS information to reserve segments of the ATC to prioritize different QoS categories. For example, the NIC may set bandwidth for certain types of applications and design ATC entry replacement policies accordingly. In one embodiment, a QoS policy that allows the discarding of traffic for a certain application causes the NIC to cancel the reservation of ATC entries associated with this traffic (if any) because the QoS policy allows the discarding of those packets. In another embodiment, if remote direct memory access (RDMA) or HPC traffic is running on a group of VNICs and PASIDs, which are designated as no-discard traffic, the NIC may reserve a certain percentage of ATC for this group of VNICs and PASIDs. If there is contention for the reserved portion of the ATC between this group of VNICs and PASIDs, the NIC may apply a difference weighted round-robin (DWRR) replacement or other suitable cache replacement algorithm to select entries in the ATC for eviction. More refined QoS policies may also be applied based on the capabilities of the NIC (such as QoS policies based on traffic categories, flow matching, packet matching, deep packet inspection, etc.).

[0026] In some embodiments, the NIC may receive a rate limit and determine an ATC entry replacement policy based on the rate limit. For example, if a rate limit is set for a group of VNICs, the NIC may use the same cache replacement algorithm (e.g., DWRR, Least Recently Used (LRU), etc.) to determine whether ATC entries associated with the group should be prefetched and / or whether to immediately invalidate ATC entries associated with the group. As another example, if the rate limit is set very low, the NIC may abandon requesting the ATC for the translation address of the group of VNICs. More refined rate limiting policies may also be applied based on the capabilities of the NIC (such as rate limiting policies based on traffic class, flow matching, packet matching, deep packet inspection, etc.).

[0027] In some embodiments, the NIC may evaluate the characteristics of the workload policy to determine the ATC entry replacement policy. For example, the NIC may monitor the traffic of a set of VNICs and use counters to quantify the number of hits for each ATC entry. The NIC may then determine an eviction scheme based on the counts. Depending on the capabilities of the NIC, more refined workload-based policies may also be applied. For example, the NIC may support network monitoring capabilities and these capabilities may be leveraged to create more precise workload-based policies. Alternatively or in addition, the NIC may adjust the ATC to take advantage of any of the above strategies to provide optimal throughput for various workloads.

[0028] In this way, certain classes of traffic can receive the highest quality of service while the computing system can improve utilization of IOTLBs (e.g., IOMMU TLBs and ATCs) and other resources. For example, the depth of the ATC, the cost of the computing system, and the power consumption can be directly related to each other; a deeper ATC can correspond to higher computing system cost and greater power consumption. Therefore, improving the utilization of the ATC can reduce hardware and energy costs. Many other features and advantages that can be provided according to various embodiments will be described and suggested below.

[0029] Figure 1 A block diagram of an example of a network 100 for deploying various embodiments of the present disclosure is shown. One of ordinary skill in the art will appreciate that there may be more or fewer nodes, devices, links, networks, or components of similar or alternative configurations for the network 100 and any system discussed in the present disclosure. Various embodiments may include different numbers and / or types of clients, networks, nodes, cloud components, servers, software components, devices, virtual or physical resources, configurations, topologies, services, devices, deployments, network devices, etc. For the sake of brevity and clarity, illustrations and examples are provided in the present disclosure.

[0030] In this example, network 100 includes a dedicated wide area network (WAN) 102 (e.g., dark fiber, multi-protocol label switching (MPLS), Software Defined-WAN (SD-WAN) structure, etc.), a first private local area network 104A (e.g., Cisco Software Defined Access (SD-Access) structure, Application Centric Infrastructure (Cisco ACI TM ) structure, Cisco hyper-convergence TM HX data platform, etc.) and a second private network or management network 104B (collectively referred to as 104), a public WAN 106 (e.g., the Internet, Long Term Evolution (LTE), etc.), network devices 108A and 108B (collectively referred to as 108), and a server chassis 110. The server chassis 110 is a computing infrastructure device that can be used to interconnect physical servers of various form factors (e.g., rack servers, blade servers, and other high-density server configurations) with other network elements. The server chassis 110 can provide power, cooling, connectivity, and mechanical support for physical servers (such as servers 112A and 112b (collectively referred to as 112) and network devices (such as switches, routers, network devices (e.g., load balancers, deep packet inspectors, firewalls, etc.), port extenders (e.g., port extenders 114A and 114B (collectively referred to as 114)), etc. An example of an implementation of the server chassis 110 is the Cisco Unified Computing System TM (Cisco ) blade server chassis. Server chassis 110 may include a plurality of slots (e.g., 8 half-width slots, 4 full-width slots, or other capacities) for receiving servers 112. Server chassis 110 may reduce the number of physical components and the amount of cabling relative to traditional rack or blade systems, integrate with existing infrastructure for centralized management, and operate more efficiently in terms of energy consumption than traditional systems.

[0031] In network 100, server 112A may be an example of a half-width or half-slot server, while server 112B may be an example of a full-width or full-slot server. Other embodiments may utilize servers having other types of form factors, including some embodiments having servers that do not require a chassis. For example, various embodiments may include a server as a standalone device that is communicatively coupled to server chassis 110 or network device 108. Various types of interconnects and buses may provide communicatively coupling, including any wired or wireless interconnect line, network, connection, bundle, single bus, multiple buses, crossbar network, single-stage network, multi-stage network, or other conductive medium operable to transmit data between portions of a computing device or between computing devices.

[0032] The server 112A may include a NIC 116A, a baseboard management controller (BMC) 118A, a CPU 120A, a main memory 122A, an I / O subsystem 124A, a local storage device 126A, and a boot subsystem 128A. The NIC 116A may connect the server 112A to other separate and discrete network elements (e.g., NICs 116B and 116C, port expanders 114, network devices 108, networks 102, 104, and 106, etc.) and logically separate elements of the server 112A (e.g., virtual machines, containers, or other virtual partitions of a physical server) (collectively referred to as virtualized servers). One of ordinary skill will appreciate that some of these elements may be combinable (e.g., an I / O subsystem may include a network interface) or further separable (e.g., cache memory may be distinguished from main memory), but for simplicity and clarity, the server 112A may include the above components.

[0033] BMC 118A may monitor and manage the status of server 112A. BMC 118A may include a dedicated service processor (not shown) and firmware (not shown) to provide management and monitoring capabilities independent of CPU 120A. In some embodiments, BMC 118A may be accessible even when CPU 120A may be powered off or not functioning. In some embodiments, BMC 118A may support standards defined in the Intelligent Platform Management Interface (IPMI) specification. An example of one implementation of BMC 118A is Integrated Management Controller (CIMC). CIMC complies with the IPMI specification but can also provide additional functionality to provide unified monitoring and management of multiple computing systems. For example, CIMC can provide diagnostic and health monitoring capabilities, such as support for Simple Network Management Protocol (SNMP); Extensible Markup Language (XML) Application Programming Interface (API) event subscriptions and configurable alerts; system event logging; audit logs; monitoring of field replaceable units (FRUs), hard disk drive (HDD) failures, dual in-line memory module (DIMM) failures, NIC media access control (MAC) addresses, CPU, and thermal faults; configurable alerts and thresholds; watchdog timer; redundant array of independent disks (RAID) configuration and monitoring; predictive failure analysis of HDDs and DIMMs; support for converged network adapters (CNAs); and support for Network Time Protocol (NTP).

[0034] In some embodiments, BMC 118A can operate in standalone mode to provide users with complete control over the server, allowing administrators to perform server management tasks, including powering on, powering off, restarting, resetting, and shutting down the server; toggling the locator light emitting diode (LED); configuring the server boot order; viewing server properties and sensors; configuring out-of-band storage; managing remote presence; managing firmware; creating and managing local user accounts and enabling authentication through Active Directory and Lightweight Directory Access Protocol (LDAP); configuring network-related settings, including NIC properties, Internet Protocol (IP) version 4 (IPv4), IP version 6 (IPv6), virtual local area networks (VLANs), and network security; configuring communication services, including Hypertext Transfer Protocol (HTTP), Secure Shell (SSH), and IPMI over a local area network (LAN); managing certificates; configuring platform event filters; and monitoring faults, alarms, and server status.

[0035] In some embodiments, the BMC 118A may also provide features such as Hypertext Markup Language Version 5 (HTML5) and Keyboard, Video, and Mouse (KVM) user interfaces (UIs); Redfish support; and XML API transaction support. HTML5 and KVM may provide a simplified UI for users and may eliminate the need for Java to use the CIMC. Redfish is an open industry standard specification and schema that specifies a Restful Stateful Transfer (REST) ​​API and uses Javascript Object Notation (JSON) and Open Data Protocol (OData) to help customers integrate the solution into their existing tool chains. XML and REST API transaction support may configure multiple managed objects in a single transaction, resulting in faster and simpler deployment.

[0036] In some embodiments, BMC 118A may perform configuration and management services while server 112A is in a low power state, such as a standby state. In contrast, CPU 120A, main memory 122A, local storage 126A, etc. may require server 112A to be in a relatively high power state. Generally speaking, a low power state may include a state in which server 112A may not be fully powered on and may not provide all or substantially all of its complete functionality, while a high power state may be a state in which server 112A may be powered on and may provide all or substantially all of its functionality, with less functionality being specifically disabled for management and configuration purposes.

[0037] CPU 120A may be connected to other elements of server 112A via one or more interconnects or buses and may directly execute instructions stored in main memory 122A and indirectly execute instructions stored in local storage device 126A, Cisco ACI TM Data center, Cisco hyper-convergence TM The CPU 120A may include any combination of hardware, software, and / or firmware that provide programmable logic. Examples of implementations of the CPU 120A may include an Advanced Reduced Instruction Set Computer (RISC) machine (ARM), a microprocessor with no interlocked pipeline stages (MIPS), PowerPC, Scalable Processor Architecture (SPARC), or a processor from Intel Corporation (Intel ) or AdvancedMicro ) Provided x86 processors, etc.

[0038] The main memory 122A may include a set of random access memories (RAM) or integrated circuits (ICs) that typically allow access to data stored in the IC in any order in a constant time, regardless of the physical location of the data. RAM may include static RAM (SRAM); dynamic RAM (DRAM); and synchronous DRAM (SDRAM). SRAM can typically be very fast, but typically has a smaller capacity than DRAM (e.g., several megabytes). SRAM can be characterized as static because they can have a chip structure that retains data as long as the SRAM has power. However, SRAM may not typically be large enough to be used as the main memory of a server. In contrast, main memory typically includes DRAM. DRAM can store data on capacitors within an integrated circuit. DRAM can be characterized as dynamic because capacitors discharge over time due to leakage current and may need to be recharged to avoid data loss.

[0039] SDRAMs have synchronous interfaces, meaning that their operation can be synchronized with a clock signal. The clock can drive an internal finite state machine to "pipelined" memory accesses (e.g., the SDRAM can accept a new memory access before the previous memory access has completed). Pipelining can improve the performance of SDRAM compared to traditional DRAM. The first implementation of SDRAM, single data rate (SDR), specifies the transfer of a single unit of data per clock cycle. The next implementation of SDRAM, double data rate (DDR), can achieve nearly twice the bandwidth of SDR by transferring data on both the rising and falling edges of the clock signal without increasing the clock frequency. DDR evolved into second generation DDR (DDR2), third generation DDR (DDR3), fourth generation DDR (DDR4), and so on. DDR2 was able to run the external data bus at twice the data rate of DDR through improved bus signaling. DDR3 improved on DDR2 by reducing operating voltage, increasing memory density, and increasing memory bandwidth. DDR4 can include improvements to power consumption and data transfer speeds.

[0040] In some embodiments, main memory 122A may include multiple memory chips assembled together on a small board called a dual in-line memory module (DIMM). DIMMs may have ranks, which is an arrangement of chips that can produce a specified number of bits of useful data (e.g., 64 bits, 128 bits, etc.). Thus, DIMMs may be single-rank, dual-rank, quad-rank, etc. The memory controller may select memory ranks by chip selects rather than address bits. For example, a memory controller may include up to eight independent chip selects and may be able to support up to eight ranks.

[0041] SDRAM DIMMs can include unbuffered DIMMs (UDIMMs) and registered DIMMs (RDIMMs). In UDIMMs, the memory chips can be directly connected to the address and control buses without any intermediate components. RDIMMs can have additional components, registers, placed between the input address and control buses and the SDRAM components. These registers can add a delay of one clock cycle, but can reduce the electrical load on the memory controller and allow more DIMMs per memory controller.

[0042] The I / O subsystem 124A may include peripheral devices and interfaces for connecting the peripheral devices to the server 112A. The I / O subsystem 124A may typically transfer data from the main memory 122A to accessible peripheral devices, and vice versa. Historically, computing systems have provided I / O using a bus compatible with the Peripheral Component Interconnect (PCI) standard, which was developed to interconnect peripheral devices to computing systems. Various embodiments may also support PCIe. PCIe may specify point-to-point connections, resulting in a tree-like structure topology with a single root complex. The root complex may be responsible for system configuration, enumeration of PCIe resources, and interrupt and error management of the PCIe tree. The root complex and its endpoints may share an address space and communicate through memory reads, writes, and interrupts. PCIe may connect two components via a point-to-point link. The link may include N channels (e.g., a x16 PCIe link may include N channels), and each channel may include two pairs of wires: one pair for sending and one pair for receiving. Each lane can be connected to a PCIe endpoint, a PCIe switch, or a PCIe-to-PCI bridge.

[0043] The local storage device 126A may include non-volatile memory and may be a hard disk drive (HDD), a non-volatile memory express (NVMe) drive, a solid-state device (SSD), or other type of computer-readable medium that can store data accessible by the computing system, such as a universal serial bus (USB) flash drive, a flash memory card or stick, an optical disk, a magnetic tape, a stand-alone RAM disk, a read-only memory (ROM), and mixtures thereof.

[0044] The boot subsystem 128A may include software and / or firmware for performing hardware initialization when the server 112A is powered on or started, and provides runtime services for operating systems and applications. The boot subsystem 128A may initialize and test the hardware components of the server 112A. The boot subsystem 128A may also load one or more operating systems from the local storage device 126A, the network 102, 104 and 106, or other storage locations into the main memory 122A. In addition, the boot subsystem 128A may discover and set one or more peripheral devices for access by the CPU 120A. In some embodiments, the server 112A may store the boot subsystem 128A as one or more boot images in a peripheral memory device connected to the server 112A. Alternatively or additionally, the network 102, 104 or 106 may store one or more boot images, from which the server 112A may retrieve the boot image. Multiple boot images may represent different hardware, software (e.g., operating system, application, etc.) and / or firmware configurations of the server 112A. Examples of implementations of the boot subsystem 128A may include a basic input / output system (BIOS) for the x86 architecture or a unified extensible firmware interface (UEFI) or a bootloader for the ARM architecture.

[0045] In some embodiments, BMC 118A may program and / or execute boot subsystem 128A to configure two physical Ethernet links to combine them into a dual-capacity Ethernet interface that can shield link failures (at the expense of half the capacity); configure the Ethernet link to split it into an out-of-band management Ethernet interface dedicated to BMC 118A with its own MAC address and one or more in-band Ethernet interfaces with different MAC addresses; configure a group of disks to organize them into a RAID configuration to form one or more fault-tolerant disks; configure PCIe devices to expose or hide them from one or more operating systems of server 112A; and monitor and manage power, voltage, clock, CPU speed, temperature, and fans.

[0046] In some embodiments, the BMC 118A may also program and / or execute the boot subsystem 128A to configure the memory controller so as to restrict access to certain portions or ranges of the main memory 122A to certain processors of the CPU 120A. For example, the server 112A may include multiple memory controllers and multiple processors (e.g., multiple sockets and / or multiple cores for each processor), and the BMC 118A may program and / or execute the boot subsystem 128A to restrict access to certain portions of memory by certain processors, thereby effectively dividing the memory between the processors. Alternatively or additionally, the BMC 118A may program and / or execute the boot subsystem 128A to restrict access to certain portions of memory by certain cores and to control whether the cores can interrupt each other. In some embodiments, the BMC 118A may also program and / or execute the boot subsystem 128A to control whether certain virtual or physical peripherals are accessible to a particular processor or core. In some embodiments, the BMC 118A may load a different boot image for each partition of a processor and / or core so that each partition can be independently booted. In this way, BMC 118A can create multiple isolated resource groups, each of which includes one or more processors or cores, memory ranges, and one or more accessible peripherals (and / or PCIe address ranges). Thus, BMC 118A can perform operations similar to traditional hypervisors without the additional overhead that may come with operating a host operating system and a guest operating system. This approach can also be an improvement over containerization because server 112A is not necessarily limited to a single operating system.

[0047] The full-slot server 112B may include NICs 116B and 116B, BMC 118B, CPU 120B, main memory 122B, I / O subsystem 124B, local storage 126B, and boot subsystem 128B. One of ordinary skill in the art will appreciate that the full-slot server 112B may be similar to the half-slot server 112A in many respects. However, the full-slot server 112B may include more computing resources than the half-slot server 112A. For example, the full-slot server 112B may include more processors (and / or cores), more DIMMs, and more I / O interfaces (including network interfaces), thereby providing greater processing power, memory, and networking capabilities relative to the half-slot server 112A.

[0048] The server 112 can be connected to the port extender 114 via the NIC 116a. The port extender, standardized by the Institute of Electrical and Electronics Engineers (IEEE) 802.1Qbh protocol, can be used as an access device for NICs, blade switches, top-of-rack (TOR) switches, hypervisors, SR-IOV adapters, virtual switches, etc. The port extender can be attached to the MAC port of an 802.1Q bridge (e.g., a control bridge) to provide additional MAC ports (downlink interfaces) that can operate as logical ports of the 802.1Q bridge to which the port extender is attached. Packets can flow from the port extender through the control bridge to enable consistent forwarding of traffic and policy enforcement. The control bridge can operate as a logical centralized management point for its collection of port extenders. Examples of implementations of the port extender 114 may include Fabric Extender (FEX) technology, such as the Cisco TOR switches used to provide additional ports Fabric Extender (FEX), used for Cisco Blade server chassis provides additional ports for Cisco Fabric expanders, used to provide additional ports for servers Adapter Fabric Extender (Adapter FEX), and to provide additional ports for virtual machines Data Center Virtual Machine Fabric Extender (VM-FEX).

[0049] Each port extender 114 may include an interconnect infrastructure I, a chassis manager M, and a chassis management switch S. The interconnect infrastructure I may operate as a bridge between the server 112 and the switch / router 108 and implement the data plane of the port extender 114. Examples of implementations of the interconnect infrastructure I may include FEX application-specific integrated circuits (ASICs), such as Redwood and Woodside. The chassis manager M can interact with the global network manager N in the switch / router 108 and the BMC 118 in the server 112. The chassis manager M can perform server discovery and sorting, power management, temperature monitoring, and fan control loops. In some embodiments, when there are multiple port expanders in the server chassis, such as Figure 1 As shown in the example in , the chassis managers M can form a cluster according to the high availability algorithm, where one manager is active and the other is inactive. For example, there can be a serial interface between the chassis managers M for receiving heartbeats between the two managers. Failover may occur due to failure to detect a heartbeat or unplugging the active chassis manager. The network manager N can also force a failover. Examples of implementations of the chassis manager M include Chassis Management Controller (CMC) ASIC. The chassis manager switch S may provide connectivity with the BMC 118 present on the server 112. Examples of implementations of the chassis manager switch S may include Chassis Management Switch (CMS) ASIC.

[0050] Port expanders 114 can connect server chassis 110 to switches / routers 108A and 108B (collectively 108). Switches / routers 108 can operate as spine switches in a spine-and-leaf topology; aggregation / distribution switches and / or core switches in a three-layer, multi-layer, or fat-tree topology; Level 1+ switches in a BCube network topology; or other switches / routers in a suitable network topology (e.g., DCell, CamCube, FiConn, Jellyfish, Scafida, etc.). Examples of implementations of switches / routers 108 may include Cisco Fabric Interconnect, Cisco Catalyst Switch, Cisco switch, Industrial Ethernet switches, Cisco Meraki TM or Switches / routers, Integrated Services Router (ISR), Network Convergence System (NCS) routers, Aggregation Services Router (ASR), Industrial ISR, and Connect mesh routers and more.

[0051] One or more of the switches / routers 108 may include a port controller ASIC P, a crossbar fabric ASIC X, and a full network manager agent N. The port controller ASIC P may control a small set of ports (e.g., 4, 8, 16, etc.) for processing packets at ingress or egress for the managed ports. The port controller ASIC P may also handle forwarding decisions between ports. Examples of implementations of the port controller ASIC P may include Unified Port Controller (UPC) ASIC.

[0052] The crossbar fabric ASIC X may operate as a bridge between the port controllers P and may manage packet switching and scheduling. That is, the crossbar fabric ASIC X may couple the port controller of an ingress port to the port controller of an egress port to enable traffic between the ports. Examples of implementations of the crossbar fabric ASIC X may include Unified Crossbar Fabric (UCF) ASIC.

[0053] The network-wide manager agent N and network management devices located in the management network 104B (collectively referred to as the network management system) may include hardware, software, and / or firmware for monitoring and managing the servers, networks, and storage infrastructure of the network 100. The network management system may provide servers, structures, storage resources and perform device discovery, inventory, configuration, diagnosis, monitoring, fault detection, auditing, and statistics collection. For example, the network management system may automatically detect, inventory, manage, and provision system components added to or changed in the network 100. The network management system may also manage clustering, switch redundancy, and otherwise ensure high availability of servers, networks, data centers, and / or storage resources in remote networks or clouds. Examples of implementations of the network management system may include Digital Network Architecture (DNA TM )center, SD-WAN vManage, Cisco ACI TM , Application Policy Infrastructure Controller (APIC), Cisco Meraki TM or Control Panel, Cisco One TM 、Cisco Intersight TM 、Cisco Central, Cisco Manager, Cisco Guide, Integrated Management Controller (IMC) hypervisor, Cisco's original Cisco Cloud Center Container Platform, Intelligent automation cloud platform, Interconnected cloud architecture, Network Services Manager, Cisco's original Network Services Controller, Virtualization Infrastructure Manager, Data Center Network Manager, Dynamic structure automation, Structure Manager, Fog Director, Cisco Network Director Cisco Firewall Management Center, Defensive Coordinator, Security manager, or other system for monitoring and managing multiple servers, network fabrics, and / or server storage.

[0054] The switch / router 108 may support various types of traffic and include various types of ports and port controllers for connecting the server 112 to other networks, such as networks 102, 104, and 106, or any communications platform operable to exchange data or information within or between computing systems (e.g., the Internet, an ad hoc local area network, a packet data network (PDN), a LAN, a metropolitan area network (MAN), a WAN, a wireless local area network (WLAN), a virtual private network (VPN), an intranet, or any other suitable architecture or system that facilitates electronic communications). For example, the switch / router 108 may include multiple 40 Gbps Quad Small Form Factor Pluggable (QSFP) or QSFP+ ports that may operate at a native 40 Gbps speed or may operate as four 10-Gbps ports for handling Ethernet / IP traffic (e.g., traffic to and from a LAN) (e.g., by inserting a QSFP-to-4 Small Form Factor Plus Pluggable (SFP+) breakout cable), Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE) ports for handling block storage traffic (e.g., traffic to and from a storage area network (SAN)), and serial ports for handling management traffic (e.g., traffic to and from the self-management network 104B), as well as inter-process communication (IPC) (e.g., high availability, clustering, virtualization platform services, etc.).

[0055] Figure 2A and Figure 2B A block diagram of an example of computing systems 200 and 250 for implementing various embodiments of the present disclosure is shown. In this example, computing systems 200 and 250 may be respective implementations of servers 112A and 112B. The present disclosure provides additional details about each component in computing systems 200 and 250 in more detail below. However, those skilled in the art will appreciate that computing systems 200 and 250 are possible configurations, and other configurations with more or fewer components are also possible.

[0056] exist Figure 2AIn the example, computing system 200 may be an example of a half-slot two-socket server including processors 202A and 202A (collectively referred to as 202). A socket is a mounting / interconnection structure for mounting a processor on a printed circuit board (PCB) or a motherboard of a computing system. Multiple sockets may customize a server motherboard by installing processors using different clock speeds and / or power sizes. Each processor 202 may include one or more cores (e.g., 2, 4, 6, 8, etc.) (not shown), each core may replicate a basic CPU. Each core may be associated with a level 1 (LI) cache (not shown). A cache is a small, fast memory that reduces the average time to access main memory. The cores typically share a larger level 2 (L2) or level 3 (L3) cache, a bus or interconnect interface, and an external wafer connection. The number of processors in a computing system is the product of the number of sockets and the number of cores in each socket. For example, computing system 200 includes two sockets, and each socket may include four cores, for a total of eight processors. Examples of implementations of processors 202A or 202B may include Intel Core TM 、Intel AMD Epyc TM 、AMD Opteron TM 、AMD Ryzen TM 、AMD Athlon TM processor, or other server processor.

[0057] Processors 202 may be connected to each other and to an I / O hub (IOH) 206 (sometimes also referred to as a chipset) via a QuickPath interconnect (QPI). QPI may include multiple high-speed unidirectional links for interconnecting processors 202 and IOH 206 or similar computing system elements, and integrate multiple distributed memory controllers for multiple cores of processor 202. The cores within a socket may share an integrated memory controller (IMC) (not shown) with multiple memory interfaces (i.e., memory buses). The IMC may have various external connections, such as DDR3 or DDR4 memory (or other suitable memory) channels for connecting processor 202 to main memory 204 (e.g., DDR3, DDR4, or other suitable memory). IMCs and cores in different sockets may communicate with each other using QPI. In some embodiments, processor 202 may also have full access to the memory of each other processor of computing system 200 while maintaining cache coherence using a cache coherent non-uniform memory architecture (NUMA). A computing system implementing QPI may use an internal crossbar router in a socket to implement global memory accessibility (or accessibility of memory from one socket to another socket). Other embodiments may use other types of buses or interconnects to implement the interconnect, such as a front side bus (FSB), a dual independent bus (DIB), a dedicated high speed interconnect (DHSI), etc.

[0058] IOH 206 may connect processor 202 to I / O controller hub (ICH) 208 (sometimes also referred to as a south bridge) using a direct media interface (DMI) interconnect and PCIe endpoints 210A and 210B (collectively 210) using a PCIe interconnect. Examples of implementations of IOH 206 may include as well as Chipset. In this example, PCIe endpoint 210A can be a memory controller that connects computing system 200 to storage devices (e.g., HDD, NVMe drive, SSD, etc.) (not shown). The memory controller can support serial attached small computer system interface (SAS), serial advanced technology attachment (SATA), PCIe or other data transfer rates, and integrate mirroring and striping functions to provide different RAID availability levels for internal drives. PCIe endpoint 210B can be a NIC for exchanging network data between computing system 200 and other network devices (e.g., port expander 114, switch / router 108, etc.). Other types of PCIe endpoints can include graphics cards, graphics processing units (GPUs), Fibre Channel host bus adapters, and other expansion cards (add-in cards).

[0059] In some embodiments, the IOH 206 may include an IOMMU (not shown) to facilitate directory memory access (DMA) transactions for the PCIe endpoint 210. The IOH 206 may also include one or more IOTLBs (not shown) to reduce latency in address resolution. In addition, the IOH 206 may include remapping hardware (not shown) for supporting I / O virtualization translation of I / O transactions.

[0060] ICH 208 can connect the computing system 200 to relatively low-speed peripheral devices (not shown) (e.g., peripheral devices slower than PCIe endpoint 210), BMC 212, and boot subsystem 214. In this example, ICH 208 can be connected to BMC 212 via a PCIex4 link. BMC 212 can provide management access to the computing system 200 before loading the operating system, and can operate as an aggregation point for server hardware. In some embodiments, BMC 212 can have two integrated Ethernet connections (not shown) or other out-of-band channels that are redundantly connected to the management component of the access device (e.g., chassis management switch S). ​​Boot subsystem 214 may include software or firmware (e.g., BIOS, UEFI, boot loader, etc.) for initializing the hardware when the computing system 200 is powered on or started, or other management software and / or firmware executed before loading one or more operating systems of the computing system 200.

[0061] exist Figure 2B , computing system 250 can be an example of a full-slot four-socket server including processors 252A, 252B, 252C, and 252D (collectively referred to as 252). Each processor 252 can include one or more cores (e.g., 4, 6, 8, etc.) (not shown), each of which can represent a discrete processing element. In this example, each processor 252 can include 4 cores, so that the CPU of computing system 250 includes a total of 16 processors. Processors 252 can be fully interconnected to each other via QPI. The processors can also be connected to IOH 256 via QPI. In other embodiments, processors 252A and 252B can be connected to a second symmetric IOH, enabling such a computing system to include additional memory and / or PCIe slots.

[0062] Each processor 252 may also be connected to one or more serial memory buffers (SMBs) 266 using a serial memory interface (SMI) link. Each SMB 266 may be connected to one or more DIMMs (e.g., DDR3, DDR4, etc.) (collectively referred to as main memory 254). In one embodiment, the computing system may include four sockets, each socket connected to four SMBs, and each SMB connected to two DIMMs, providing a total of 32 DIMMs.

[0063] IOH 256 may be connected to PCI endpoints 260A, 260B, and 260C (collectively 260) via a PCIe interconnect. In this example, PCI endpoint 260A may be a memory controller connected to IOH 256 via a PCIe x4 link, while PCI endpoints 260B and 260C may be NICs connected to IOH 256 via a PCIe x16 link. IOH 256 may also be connected to ICH 258 via an enhanced function interface (EFI) bus. ICH 258 may provide connections to BMC 262 and boot subsystem 264. One of ordinary skill in the art will appreciate that computing system 250 may be similar to computing system 200 in many respects. However, computing system 250 is an example of a full-slot server that may include more overall resources than the half-slot server of computing system 200. For example, computing system 250 includes more sockets and cores than computing system 200, and therefore, the CPU of computing system 250 may include more processor elements, more DIMMs or other main memory, and more I / O interfaces, etc., than the CPU of computing system 200. Therefore, computing system 250 may have greater processing power, memory, and networking capabilities than computing system 200. Although computing systems 200 and 250 illustrate examples of an x86 architecture, other embodiments may utilize other server architectures (e.g., ARM, MIPS, powerPC, SPARC, other RISC architectures, and / or other complex instruction set computer (CISC) architectures), and those of ordinary skill in the art may readily apply the principles disclosed in the present invention to these other architectures.

[0064] Figure 3A A block diagram showing an example of a reference architecture of a computing system 300 for implementing I / O virtualization (IOV) and ATS, such as in PCI As described in the base specification. IOV can provide a mechanism by which a single physical function (PF) or physical device can appear as multiple independent physical functions (e.g., virtual functions (VF)) or devices (e.g., virtual devices (VDEV)), each with its own configuration, including bus numbers and base address registers (BARs), or more granular pre-allocated device resources (e.g., allocatable device interfaces (ADIs) associated with a PASID). IOV can enable a single PF or physical device to be securely shared between multiple system images (e.g., virtual machines, containers, or other virtual partitions of the computing system 300). IOV can also enable system images to interact directly with VFs or VDEVs and bypass intermediate virtualization layers (e.g., operating systems, hypervisors, container engines, or other virtualization software) to improve I / O performance.

[0065] In this example, computing system 300 (e.g., computing systems 112, 200, and 250) includes a CPU 302 (e.g., CPUs 120, 202, and 252), a main memory 304 (e.g., main memory 122, 204, or 254), a PCIe root complex 306 (e.g., IOH 206, 256), root ports 308A and 308B (collectively 308), one or more PCIe switches 310 (e.g., ICH 208 and 258), and PCIe endpoints 312A, 312B, 312C, and 312D (e.g., PCIe endpoints 116, 210, and 260) (collectively 312). Root complex 306 may represent the root of an I / O hierarchy that connects CPU 302 and main memory 304 to I / O resources (e.g., PCIe endpoint 312). The computing system may include one or more root complexes, and the root complex may include at least one host bridge, root port, or integrated endpoint (e.g., integrated PCIe endpoint 312A). Each root port 308 may correspond to a separate hierarchical domain, and each hierarchical domain may include a single PCIe endpoint 312 (e.g., PCIe endpoint 312B) or a sub-hierarchy including one or more PCIe switches 310 that provide I / O fan-out and connectivity to one or more PCIe endpoints (e.g., PCIe endpoints 312C and 312D). The PCIe endpoints 312 may be integrated within the root complex 306 (e.g., PCI endpoint 310A), or they may be distinct components separate from the root complex 306 (e.g., PCI endpoints 310B, 310C, and 310D).

[0066] In some embodiments, the computing system 300 can execute multiple system images 314A, 314B, ..., 314N (collectively referred to as 314) simultaneously. The system image 314 can be software, such as a virtual machine, an operating system, a container, or other virtual partitions of a physical server, which can be used to execute applications or services (e.g., I / O device drivers). The computing system 300 may also include a virtualization intermediary 316 (e.g., an operating system, a hypervisor, a container engine, or other virtualization software) located between the CPU 302 and the main memory 304 and the system image 314. The virtualization intermediary 316 can abstract the hardware (e.g., CPU 302, main memory 304, PCIe endpoint 312, etc.) to present each system image 314 and its own virtual system. The actual hardware resources available for each system image 314 may vary based on workload or user-specific policies. Although this may be suitable for some environments, due to the additional overhead of the virtualization intermediary 316, I / O may become a bottleneck in the computing system 300. To address this issue, IOV may enable PCIe endpoints 312 and system image 314 to bypass virtualization intermediary 316 for I / O intensive operations such as DMA, other memory accesses (eg, local storage 126), interrupt handling, and the like.

[0067] One type of IOV, namely SR-IOV, can provide a single root PCI manager (SR-PCIM) 318 for configuring and managing software for PFs 320A, 320B, ..., 320N (collectively referred to as 320) and VFs 322A, 322B ..., 322N (collectively referred to as 322) in each PCIe endpoint 312. PF 320 can be a PCIe function or device that supports SR-IOV and can be accessed by system image 314, virtualization agent 316, or SR-PCIM 318. VF 322 can be a lightweight PCIe function or device that can be directly accessed by system image 314. Each PF 320 and VF 322 can be assigned a unique routing ID or PASID for routing transaction layer packets (TLPs). PCI Additional details about SR-IOV are specified in the base specification. Other embodiments may also support other IOV technologies, such as multiple 10V, Scalable IOV, Virtualization Technology for Directed I / O (VT-d), I / O Virtualization Technology etc.

[0068] The ATS may provide a mechanism to enable the system image 314 to perform direct memory access (DMA) transactions to and from the PCIe endpoint 312. This may be accomplished by using a lookup table that maps virtual addresses to physical addresses that the system image 314 is accessing in the main memory 304. The ATS may also enable the PCIe endpoint 312 to perform DMA transactions to memory locations in the computing system 300 using the same mechanism.

[0069] ATS may allow PCIe endpoints 312 to participate in the translation process and may provide ATC for memory accesses of the I / O device itself. This may offload some address resolution processing from the CPU 302 and / or the root complex 306 by distributing translations to the PCIe endpoints 312, reduce IOTLB cache thrashing, reduce performance dependency on the size of the IOMMU TLB, and optimize memory access latency by sending pre-translation requests to the root complex 306.

[0070] DMA access time can be significantly extended by resolving virtual addresses to actual physical addresses. For example, if PCIe endpoint 312 needs to access a translation table in main memory 304, the access time can be significantly longer than an untranslated access. In addition, if each I / O transaction requires multiple memory accesses (e.g., for page table walks), the memory access rate or additional overhead associated with DMA may be high. To optimize address resolution, computing system 300 may include one or more TLBs in the entity that performs address translation (such as CPU 302, root complex 306, or PCIe endpoint 312). PCI The base specification refers to the TLB in the root complex 306 or PCIe endpoint 312 as an ATC (sometimes also referred to as a device TLB) to distinguish it from the TLB of the CPU.

[0071] The computing system 300 may include various elements for supporting ATS, such as a translation agent 324, an address translation and protection table (ATPT) 326, and ATCs 328A, 328B, 328C, and 328D (collectively referred to as 328) in PCIe endpoints 312A, 312B, 312C, and 312D, respectively. The translation agent 324 may include hardware, firmware, and / or software for translating addresses within a PCIe transaction into associated physical addresses. In some embodiments, the translation agent 324 may include an ATC to reduce latency in translation table access. The translation agent 324 may also be used to pre-fetch translation addresses before DMA access. The ATPT 326 may store address translations retrieved by the translation agent 324 to process PCIe requests, such as DMA reads or DMA writes (including writing interrupt requests to memory), and other address translation services. DMA read and write requests may be translated by a combination of routing IDs (or PASIDs) and addresses included in PCIe transactions. In some embodiments, translation agent 324 may be integrated with root complex 306, the IOMMU may operate as translation agent 324 / root complex 306, and the IOMMU may be integrated in an IOH (eg, IOHs 206 and 256).

[0072] The ATC may reside in two locations within the computing system 300, within the translation agent 324 (which may be integrated within or located above the root complex 306), or within a PCIe endpoint (e.g., ATC 328). Within the root complex 306, an ATC (not shown) may be used to accelerate translation lookups. Within the PCIe endpoint 312, the ATC 328 may be populated by the ATS and may be used in PCIe transactions to bypass the IOMMU's IOTLB to improve performance without compromising the benefits of the ATPT 326.

[0073] Figure 3B A block diagram of an example of a communication protocol 350 that enhances ATS is shown. One of ordinary skill will appreciate that for any process discussed herein, unless otherwise noted, there may be additional, fewer, or alternative steps performed in a similar or alternative order or in parallel within the scope of the various embodiments. The communication protocol 350 may be executed between a PCIe endpoint 312 and a translation agent 324 integrated within a root complex 306 (collectively, root complex 306). In some embodiments, an IOMMU may operate as a translation agent 324 / root complex 306, and the IOMMU may be integrated within an IOH (e.g., IOH 206 and 256).

[0074] In this example, the PCIe endpoint 312 may include an ATC 328. The ATC 328 may include one or more control registers 352 (e.g., ATS capability register, ATS control register, page request control register, page request status register, outstanding page request capacity, outstanding page request allocation, etc.) for enabling ATS configuration. The ATC 328 may also include two levels of caches 354 and 356 that the PCIe endpoint 312 may reference before determining whether to obtain a translation address from the host computing system via ATS.

[0075] If a transaction in the first level cache 354 misses, the PCIe endpoint 312 can refer to the second level cache 356 to determine whether the transaction is associated with any fixed entry in the second level cache 356 and / or which cache lines are available for reservation for the I / O transaction. For example, Table 1 below illustrates an example of an implementation of the second level cache 356. The second level cache 356 may include a VNIC identifier, a PASID, a priority, and a first level area of ​​the ATC reserved for those VNICs and PASIDs. The address, size, and count values ​​may be updated at runtime.

[0076] Table 1: Example L2 address translation cache

[0077]

[0078] The second level of the ATC can identify to the NIC the cache lines that are explicitly reserved for VNICs 10, 14, and 23, as shown in Table 1. When the NIC generates a request for a translated address through the ATS, the NIC can skip the reserved entries and update the rest of the first level of the ATC by invalidating the entries outside the reserved area. This design ensures that there is room in the first level of the ATC for non-priority transactions, which can be marked as dirty and invalid so that non-priority traffic is not permanently blocked, or otherwise provide non-blocking I / O to non-priority traffic. In any case, if there are not enough cache lines in the first level of the ATC, sending untranslated transactions to the host computing system may be less harmful, just as it is for low-priority network flows. Therefore, the ATC is highly adaptable and configurable for a variety of hyperscale workloads.

[0079] In some embodiments, the communication protocol 350 may include a translation request 360, a translation completion 366, a translated request 370, an invalidation request 382, ​​an invalidation completion 384, a page request 390, and a page response 396. Prior to the translation request 360, the PCIe endpoint 312 may receive a work request, such as data being transferred to the PCIe endpoint 312 for egress and requiring the PCIe endpoint 312 to read it from the main memory 304, or data being received by the PCIe endpoint 312 for ingress and requiring the PCIe endpoint 312 to write it to the main memory 304. The PCIe endpoint 312 may determine that the ATC 328 has not cached a location to be read from or written to the main memory 304. The PCIe endpoint 312 may also determine that caching the translation within the ATC 328 may be beneficial, such as if the memory address range may be frequently accessed over an extended period of time or the buffer contents associated therewith may be subject to a significant update rate. In some embodiments, the translation request 360 may be generated based on one or more ATC entry replacement policies (e.g., user-defined, QoS, rate limiting, workload-based policies, etc.) and / or ATC entry replacement algorithms (e.g., Deficit Weighted Round Robin (DWRR)).

[0080] The translation request 360 may be sent upstream through the PCIe hierarchy to the root complex 306. In some embodiments, multiple translation requests 360 may be aggregated and then sent at once to reduce the additional overhead associated with each request. Upon receiving the translation request 360, the root complex 306 may reference the ATPT 326 (at 362) to verify that the PCIe endpoint 312 can access the memory indicated by the request and obtain the translated address. In this example, the ATPT 326 may be integrated into the main memory 304. The root complex 306 may receive the translated address (at 364) and send a translation completion 366 to the PCIe endpoint 312 along with the translated address. The PCIe endpoint 312 may update the ATC 328 with a mapping of the untranslated address and the translated address and continue processing the work request with the translated address.

[0081] The work request received by the PCIe endpoint 312 may similarly be preceded by the translated request 370. However, the PCIe endpoint 312 may instead determine that the ATC 328 has cached the location to be written or read in the main memory 304, and the PCIe endpoint 312 may obtain the cached translation (e.g., the result of the translation completion 366) from the ATC 328. The translated request 370 may be a regular read or write request to the main memory 304 using the cached translation, and the main memory 304 may respond with data in the case of a read request, or with an indication of a successful write in the case of a write request (at 372).

[0082] When the translation in the ATPT 326 changes (e.g., because the translation is out of date or determined to be invalid for other reasons), the root complex 306 may receive an invalidation request from software (at 380) and send an invalidation request 382 to the ATC 328 to maintain proper synchronization between the ATPT 326 and the ATC 328. The invalidation request 382 may be used to clear a specific subset of the address range from the ATC 328. After the PCIe endpoint 312 completes the invalidation operation (at 384), the PCIe endpoint 312 may send an invalidation completion 386.

[0083] Prior to the page request 390, the PCIe endpoint 312 may determine that it needs to access a page for which the ATS translation is not available. The PCIe endpoint 312 may cause the page request 390 to be sent to the root complex 306 (e.g., via a request to a page request interface). The page request 390 may include a page address and a page request group (PRG) index. The PRG index may be used to identify transactions and match requests with responses.

[0084] After the root complex 306 determines its response to the request (e.g., fetches the page from the ATPT 326 at 392 and makes the requested page resident at 394), the root complex 306 may send a page response 396 back to the PCIe endpoint 312. The PCIe endpoint 312 may then use the ATS to request a translation of the requested page.

[0085] Figure 4 A block diagram of an example of a NIC 400 that can be used to implement one embodiment of the present disclosure is shown. In this example, the NIC 400 includes a PCIe / media access control (MAC) interface 312, a device controller 402, a transmit direct memory access (DMA) engine 404, a receive DMA engine 406, a device memory 408, and a network buffer 410, each of which can be interconnected by a crossbar 412. The PCIe port 312 can include an ATC 328 and can be connected to the root port 308 via a PCIe x16 link.

[0086] DMA engines 404 and 406 can be instructed by device controller 402 to read and write data between local memory (e.g., device memory 408, network buffer 410) and memory of the host computing system. In some embodiments, DMA engines 404 and 406 can offload various network and packet processing tasks from the CPU of the host computing system, such as Remote DMA over Converged Ethernet (RoCE) offload; Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) stateless offload or other network protocol processing; Large Send Offload (LSO), Large Receive Offload (LRO), Checksum Offload; Receive Side Scaling (RSS), Send Side Scaling (TSS), Virtual Extensible LAN (VXLAN) or MPLS labeling, receive flow steering; Data Plane Development Kit (DPDK), Open Virtual Switch (OVS) offload, flow table matching, tunnel encapsulation / decapsulation, and encryption / decryption, etc. The device memory 408 may include a DDR interface for adding additional device memory and may be used for temporary storage of frames, buffer descriptors, and other control data.

[0087] The NIC 400 may also include an Ethernet / MAC interface 414, which is connected to the network buffer 410 at one end and connected to one or more network devices (e.g., port expander 114, switch / router 108, etc.) (not shown) at the other end through one or more Ethernet links (e.g., 1GbE, 10GbE, 25GbE, 40GbE, 50GbE, 100GbE, 200GbE, etc.). The Ethernet / MAC interface 414 may interact with the device controller 402 to receive frames into the network buffer 410 and send frames from the network buffer 410 to the network.

[0088] In some embodiments, NIC 400 may be a programmable NIC and include one or more programmable elements (eg, ASIC, field programmable gate array (FPGA), etc.) that may run compiled code firmware.

[0089] The NIC may use the ATC to obtain a translated physical address of a host computing system (e.g., computing system 112, 200, 250, 300, etc.) for a given untranslated address to reduce memory access latency that may be required to retrieve the translated address for a given I / O transaction. Depending on the size of the payload supported by the NIC, the latency of the transaction may vary in the presence of ATC jitter. For example, if the NIC only supports a maximum payload of 256B and must transfer 9KB of data to its host computing system (e.g., a virtualized system with IOMMU enabled), the NIC may send up to 36 transactions to the host computing system under maximum load. Each transaction may involve a write to an untranslated address in the main memory of the computing system, which may require the IOMMU to determine the physical address mapped to the untranslated address. If the address is not cached and does not appear in the IOTLB, the IOMMU may need to traverse the page table to determine the corresponding physical address, which the NIC can use to write the payload to the target address. This can cause a significant degree of latency and reduce overall throughput even if the NIC is performing large transfers.

[0090] To reduce memory access latency, ATS can allow PCIe endpoints to merge their own ATCs. This can improve performance to some extent for medium-level workloads, but may not be effective for very large scale (e.g., thousands). With the introduction of PASID, computing systems are expected to run more than a thousand system images (e.g., virtual machines, containers, or other virtual partitions of the computing system) simultaneously. This will put tremendous pressure on the ATC. Since streams for different system images may have different address domains and different PASIDs, traditional ATCs may be affected by cache thrashing and the entire system may experience severe throughput loss (e.g., a large number of I / O transactions may be throttled during address resolution). The utilization of ATC can be optimized by defining ATC entry replacement policies based on user preferences, QoS, rate limits, and other criteria to overcome these and other deficiencies of the prior art.

[0091] Figure 5 A flow chart of an example process 500 for optimizing utilization of an ATC for a network I / O device is shown. Process 500 may be performed at least in part by a NIC (e.g., NIC 400) and an I / O subsystem (e.g., I / O subsystem 124, IOHs 206 and 256, root complex 306, translation agent 324, IOMMU, etc.). The NIC may include an ATC (e.g., ATC 328) or a device TLB, and the I / O subsystem may include an IOTLB or an ATC.

[0092] Process 500 may begin at step 502, where the NIC may receive reservation information to reserve one or more cache lines in a first level of the ATC (e.g., a first discrete ternary content addressable memory (TCAM) or a first segment of a discrete ATC TCAM) and write the reservation information to a second level of the ATC (e.g., a second discrete TCAM or a second segment of a discrete ATC TCAM). The reservation information may include at least identification information (e.g., a VNIC identifier, a PASID, L2-L7 data, etc.) and the reserved ATC entries or cache lines. In some embodiments, the reservation information may also include priority information and other metadata for entries in the second level of the ATC.

[0093] In some embodiments, the second level of ATC can be managed by a network management system (e.g., Cisco DNA TM center, SD-WAN vManage, Cisco ACI TM , Application Policy Infrastructure Controller (APIG), Cisco Memki TM or Control Panel, Cisco Manager, etc.) based on one or more ATC entry replacement policies. The ATC entry replacement policy can be automatically generated, or can be based on user configuration. For example, a user can request the network management system to prioritize traffic based on MAC address, and the network management system can determine the VNIC identifier and PASID associated with this MAC address, and the number of cache lines in the first level of the ATC reserved for these VNIC identifiers and PASIDs. The network management system can write the reservation information to the second level of the ATC. In some embodiments, the reservation information can be located in a storage device (e.g., a local storage device of the host computing system, a local storage device of the NIC, a remote storage device, etc.), and can be retrieved and written to the second level of the ATC, such as when the host computing system starts up.

[0094] In some embodiments, the ATC entry replacement policy can be composed of QoS levels (e.g., Class of Service (CoS); IP Type of Service (ToS); IP Precedence (IPP) 0-7; Differentiated Services Code Point (DSCP) and Per-Hop Behavior (PHB), such as Best Effort (BE or DSCP 0), Internet Engineering Task Force (IETF) Request for Comments (RFC) 2474 Class Selector (CS) 1-7, RFC 2597 Assured Forwarding PHB (AFxy), RFC 3268 Expedited Forwarding (EF), IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification (IP Explicit Congestion Notification ( ECN), etc. The QoS management system or other suitable software can divide out areas of the first level of ATC to prioritize traffic according to the defined QoS levels, and write the divided parts to the second level of ATC. For example, a user can define a QoS level, allocate more of the first level of ATC, and give more weight to flows that do not drop Ethernet and Fibre Channel (FC) so that these flows do not fail when they are active at any given time. If there is contention for the first level of ATC reserved for a specific QoS level, the NIC can use LRU, DWRR, or other suitable algorithms to determine which first level ATC entries reserved for this QoS level to evict. QoS can also define a minimum amount of bandwidth to allocate to all traffic flowing through the NIC. For example, the QoS management system or other software can divide out the first level of ATC to prevent starvation due to invalidation of the first level ATC cache lines, and write the divided parts to the second level of ATC.

[0095] In some embodiments, rate limits may dictate ATC entry replacement policies. Rate limits may generally limit the amount of bandwidth that can be consumed over a period of time, and these limits may be used as criteria for partitioning the first level of the ATC. For example, low bandwidth utilization flows may be accommodated in smaller regions in the first level of the ATC. The partitioning of the first level of the ATC may be determined by software to resolve conflicting policies and prevent starvation due to static partitioning of the first level of the ATC.

[0096] After the second level of the ATC is populated with the reservation information, the process 500 may continue to step 504, where the NIC may receive an indication to read from or write to the main memory of the host computing system. For example, an ingress or egress unit of the NIC (e.g., receive and transmit direct memory access (DMA) engines 404 and 406) may generate an I / O transaction (e.g., translation request 360) or a DMA transaction to retrieve from or write to an untranslated address provided by a device controller of the NIC (e.g., device controller 402), an IOMMU of the host computing system, software running on the host computing system, etc. In one embodiment, the NIC may receive external network data to an Ethernet interface (e.g., Ethernet interface 414), which may be processed by a receive DMA engine. When processing is complete, the receive DMA engine may generate an I / O request to send the processed data to the untranslated address. In another embodiment, the egress DMA engine may receive an indication that data is available in a main memory of the host computing system, and the egress DMA engine may generate an I / O request to fetch the data using an untranslated address.

[0097] At decision block 506, the NIC may refer to the first level of the ATC to determine whether the untranslated address has been cached. For example, the ingress / egress unit may send an I / O transaction to a PCIe interface of the NIC (e.g., PCIe interface 312), which may prompt the NIC to search the first level of the ATC. If the untranslated address has been cached, the NIC may obtain the translated address from the first level of the ATC, and process 500 may proceed to step 516.

[0098] If the untranslated address is not cached in the first level of the ATC, the process 500 can proceed to decision block 508, where the NIC can determine whether there are any available entries in the non-privileged area of ​​the first level of the ATC that can be overwritten and select from those entries (if available). For example, there may be empty entries or invalid ATC entries in the first level of the ATC. If so, the NIC can select from the available entries in the first level of the ATC (e.g., random, least recently used (LRU), deficit weighted round robin (DWRR), or other suitable algorithm), and the process 500 can proceed to step 516.

[0099] If there are no available entries to write in the first level of the ATC, the NIC may proceed to decision block 510 and query the second level of the ATC to determine whether the current I / O transaction is associated with any reserved or fixed entries in the second level of the ATC. For example, the second level of the ATC may include a VNIC identifier, a PASID, and start and end cache lines indicating the portion of the first level of the ATC that has been fixed or reserved for each VNIC identifier and PASID. The NIC may utilize the VNIC identifier and PASID of the current I / O transaction to determine whether they match any entries in the second level of the ATC. In other embodiments, the second level of the ATC may store other data (e.g., L2-L7 data) for classifying traffic based on the hardware capabilities of the NIC, and the NIC may be able to retrieve the corresponding data from the current I / O transaction. In other embodiments, the NIC may utilize other identifying information of the current I / O transaction to match entries in the second level of the ATC. If the current I / O transaction matches an entry in the second level of the ATC, the NIC may select an entry in the first level of the ATC, evict from the cache line associated with the matching VNIC identifier and PASID in the second level of the ATC, and process 500 may proceed to step 512. In some embodiments, the NIC may apply a least recently used (LRU) or other suitable algorithm (e.g., a deficit weighted round robin (DWRR)) for this selection.

[0100] If the current I / O transaction is not associated with any pinned or reserved entry of the first level of the ATC (e.g., the current I / O transaction is not a privileged transaction), the NIC may select an entry in the non-privileged area of ​​the first level of the ATC for eviction, and process 500 may proceed to step 512. In some embodiments, the NIC may apply a least recently used (LRU) or other suitable algorithm (e.g., a deficit weighted round robin (DWRR)) for selection.

[0101] At step 512, the NIC may generate a translation request to retrieve the translated address of the current I / O transaction, and process 500 may proceed to step 514. At step 514, the NIC may cache information mapping the untranslated address to the translated address at the selected entry in the first level of the ATC, and the process may proceed to step 516. In other words, the NIC may evict or overwrite the selected entry of the first level of the ATC with the mapping of the untranslated address to the translated address based on the ATC replacement policy written to the second level of the ATC in step 502.

[0102] At step 516, the NIC may perform DMA using the translated address (e.g., translated request 370 or a regular read / write to host memory), and may end process 500 following an indication that the DMA was successful (e.g., translation complete 366 or other indication that the read / write to host memory was successful). In this manner, the NIC may enforce an ATC entry replacement policy to more intelligently select entries in the first stage of the ATC for eviction when necessary and ensure performance and utilization of a set of privileged VNIC identifiers, PASIDs, or other identifiable prioritized traffic.

[0103] In summary, in some embodiments, systems and methods are used to optimize the utilization of an address translation cache (ATC). A network interface controller (NIC) may write information that one or more cache lines are reserved in a first level of the ATC to a second level of the ATC. The NIC may receive a request for direct memory access (DMA) of an untranslated address in a memory of a host computing system. The NIC may determine that the untranslated address is not cached in the first level of the ATC. The NIC may use the request and the second level of the ATC to identify a selected cache line in the first level of the ATC for eviction. The NIC may receive a translated address of the untranslated address. The NIC may cache the untranslated address in a selected cache line. The NIC may perform DMA using the translated address.

[0104] For clarity of explanation, in some instances, various embodiments may be presented as including separate functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or a combination of hardware and software.

[0105] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0106] The method according to the above examples can be implemented using computer executable instructions, which are stored in or otherwise available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Some of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, and the like.

[0107] Devices implementing methods according to these disclosures may include hardware, firmware, and / or software, and may be implemented in any of a variety of form factors. Some examples of such form factors include general-purpose computing devices, such as servers, rack-mounted devices, desktop computers, laptop computers, etc., or general-purpose mobile computing devices, such as tablet computers, smart phones, personal digital assistants, wearable devices, etc. The functions described herein may also be embodied in peripheral devices or expansion cards. By way of further example, such functions may also be implemented between different chips on a circuit board or different processes executed in a single device.

[0108] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0109] Although various examples and other information are used to illustrate aspects within the scope of the appended claims, no limitation to the claims should be implied based on the specific features or arrangements in such examples, as a person of ordinary skill will be able to use these examples to derive a variety of implementations. In addition, although some subject matter may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or actions. For example, such functionality may be allocated in a different manner or performed in components different from those identified herein. Instead, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

1. A computer-implemented method comprising: receiving a request to reserve one or more cache lines for a set of network flows; determining information identifying the set of network traffic; determining to retain the one or more cache lines using, at least in part, the information identifying the set of network traffic; writing, via a network interface controller NIC connected to the host computing system, information retaining the one or more cache lines in a first level of an address translation cache ATC into a second level of the ATC; receiving a direct memory access (DMA) request for an untranslated address corresponding to a physical address in a memory of the host computing system; identifying a selected cache line in a first level of the ATC for eviction using the request and a second level of the ATC; writing the untranslated address and the physical address into the selected cache line; as well as The DMA is performed using the physical address.

2. The computer-implemented method of claim 1 , further comprising: It is determined that the untranslated address is not cached in a first level of the ATC.

3. The computer-implemented method of claim 1 or 2, further comprising: receiving a second DMA request for a second untranslated address corresponding to a second physical address in a memory of the host computing system; determining that the second untranslated address is cached in a first level of the ATC; as well as The second DMA is performed using the second physical address.

4. The computer-implemented method of claim 1 or 2, further comprising: determining that the request does not correspond to a cache line in a second level of the ATC, The selected cache line is selected from one or more unreserved cache lines in the first level of the ATC.

5. The computer-implemented method of claim 1 or 2, further comprising: Information to reserve the one or more cache lines is received from a network management system.

6. The computer-implemented method of claim 5, further comprising: receiving, by the network management system, one or more ATC entry replacement policies; and The retained information is determined by the network management system using the one or more ATC entry replacement policies.

7. The computer-implemented method of claim 1 or 2, further comprising: Receiving quality of service (QoS) information of the set of network traffic; determining a QoS level for the set of network traffic using the QoS information; as well as Information of retaining the one or more cache lines is determined using the QoS level.

8. The computer-implemented method of claim 1 or 2, further comprising: receiving a rate limit for the set of network traffic; determining bandwidth utilization information for the set of network traffic; as well as Information to retain the one or more cache lines is determined using the rate limit and the bandwidth utilization information.

9. The computer-implemented method of claim 1 or 2, further comprising: Monitoring the number of cache line hits in the second level of the ATC; and The number of hits is used to determine whether to retain information for the one or more cache lines.

10. The computer-implemented method of claim 1 or 2, further comprising: A Peripheral Component Interconnect Express (PCIe) Address Translation Service (ATS) translation request is sent using the untranslated address.

11. The computer-implemented method of claim 1 or 2, wherein: The selected cache line is identified using Deficit Weighted Round Robin DWRR.

12. A system for optimizing the utilization of an address translation cache (ATC), comprising: one or more processors; and A memory including instructions that, when executed by the one or more processors, cause the system to perform the following operations: receiving a request to reserve one or more cache lines for network traffic; determining information identifying the network traffic; determining to retain the one or more cache lines using, at least in part, the information identifying the network traffic; writing, by a network interface controller, information retaining the one or more cache lines in the first level of the ATC to a second level of the ATC; receiving a direct memory access (DMA) request for an untranslated address corresponding to a physical address in a memory of a host computing system; identifying a selected cache line in a first level of the ATC for eviction using the request and a second level of the ATC; writing the untranslated address and the physical address into the selected cache line; as well as The DMA is performed using the physical address.

13. The system according to claim 12, wherein: The information for retaining the one or more cache lines includes a virtual network interface controller VNIC identifier and a process address space identifier PASID.

14. The system according to claim 13, wherein: The first stage of the ATC includes a first segment of a discrete ternary content addressable memory (TCAM), and the second stage of the ATC includes a second segment of a discrete TCAM.

15. A system according to any one of claims 12 to 14, wherein: The network interface controller is connected to the host computing system via one or more Peripheral Component Interconnect Express (PCIe) lanes.

16. A non-transitory computer-readable storage medium comprising instructions, which, when executed by one or more processors of a system, cause the system to perform the following operations: receiving a request to reserve one or more cache lines for network traffic; determining information identifying the network traffic; determining to retain the one or more cache lines using, at least in part, the information identifying the network traffic; Writing information retaining the one or more cache lines in a first level of an address translation cache ATC of a network interface controller into a second level of the ATC; receiving a direct memory access (DMA) request for an untranslated address corresponding to a physical address in a memory of a host computing system; identifying a selected cache line in a first level of the ATC for eviction using the request and a second level of the ATC; writing the untranslated address and the physical address into the selected cache line; as well as The DMA is performed using the physical address.

17. The non-transitory computer-readable storage medium of claim 16, wherein: When the instructions are executed, the system is further caused to perform the following operations: receiving minimum bandwidth information for the network traffic; and One or more ATC entry replacement strategies are determined using the minimum bandwidth information to provide at least a minimum bandwidth to the network traffic.

18. The non-transitory computer-readable storage medium according to claim 16 or 17, wherein: When the instructions are executed, the system is further caused to perform the following operations: One or more ATC entry replacement policies are determined to provide non-blocking input / output (I / O) for the network traffic.

19. The non-transitory computer-readable storage medium of claim 18, wherein: When the instructions are executed, the system is further caused to perform the following operations: Cache lines in the first level of the ATC that do not correspond to cache lines in the second level of the ATC are invalidated.

20. A device for optimizing the utilization of an address translation cache ATC, comprising: means for receiving a request to reserve one or more cache lines for network traffic; means for determining information identifying said network traffic; means for determining to retain information for the one or more cache lines using, at least in part, information identifying the network traffic; means for writing information of one or more cache lines reserved in a first level of said ATC to a second level of said ATC via a network interface controller NIC connected to a host computing system; means for receiving a direct memory access (DMA) request for an untranslated address corresponding to a physical address in a memory of the host computing system; means for identifying a selected cache line in a first level of the ATC for eviction using the request and a second level of the ATC; means for writing the untranslated address and the physical address to the selected cache line; Means for performing the DMA using the physical address.

21. The apparatus of claim 20, further comprising means for implementing the method of any one of claims 2 to 12.

22. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 11.

Images