System and method for address translation in switch device

Abstract

The present technology relates to systems and methods for address translation in a switch device. In an embodiment, the switch device includes a first port configured to receive a first request associated with a first address, and a cache configured to store a plurality of mapping entries. The switch device further includes a routing unit coupled to the cache, the routing unit configured to determine whether a second address associated with the first address is present in the cache. The cache allows for efficient storage and retrieval of frequently used address translations, reducing the need to repeatedly access the host system for address translations. This minimizes latency in handling data requests and improves overall system performance.

Description

System and method for address translation in switching devices Technical Field

[0001] This disclosure relates to the field of data communications, and more particularly to systems and methods for address translation in switching devices. Background Technology

[0002] In modern computing and networking environments, reliable and efficient communication between devices is crucial for maintaining system performance and uptime. Many systems involve multiple devices, such as network interface cards (NICs), storage devices, and processing units, working together to handle large volumes of data. These devices can be interconnected via switches that manage data routing between the devices and external systems, including host systems and other endpoints.

[0003] Some methods for data transfer between devices rely on Direct Memory Access (DMA), which allows devices to directly access memory without increasing the burden on the Central Processing Unit (CPU). This improves overall efficiency by reducing processing overhead and enabling faster data transfers. For example, Peripheral Component Interconnect Fast (PCIe) is a standard that supports high-speed communication between devices such as NICs, processing units, and memory controllers. PCIe enables direct connections between devices via a bus architecture, thereby facilitating efficient data flow between multiple endpoints through switches.

[0004] As systems become increasingly complex, especially for high-performance workloads such as artificial intelligence (AI) and machine learning (ML), the efficiency of address translation and data routing becomes increasingly important. Devices that generate data requests (e.g., those involving DMA) typically use virtual addresses, which must be translated to physical addresses before data can be routed to its destination. In various implementations, address translation can be handled by mechanisms such as the input / output memory management unit (IOMMU). However, frequent address translation can introduce latency, especially when the same address translation is repeatedly requested, thus impacting overall system performance.

[0005] Various methods have been explored for performing address translation in complex systems, but they have proven insufficient. It is important to recognize the need for new and improved systems and methods. Summary of the Invention

[0006] One embodiment of this disclosure provides a switching device. The switching device includes: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries; and a routing unit coupled to the cache. The routing unit is configured to: determine whether a second address, associated with the first address, exists among the plurality of mapping entries; retrieve the second address in response to determining that the second address exists among the plurality of mapping entries; and forward the first request to a first device to obtain the second address in response to determining that the second address does not exist among the plurality of mapping entries, the first device being configured to provide the second address by performing address translation based on the first address. The switching device also includes: a second port coupled to the routing unit, the second port being configured to transmit the first request based on the second address; wherein the routing unit is configured to update the cache with a first mapping entry that associates the first address with the second address obtained from the first device.

[0007] Another embodiment of this disclosure provides a switching device. The switching device includes: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries; and a routing unit coupled to the cache. The routing unit is configured to: determine whether a second address, associated with the first address, exists among the plurality of mapping entries; retrieve the second address in response to determining that the second address exists among the plurality of mapping entries; and forward the first request to a first device to obtain the second address in response to determining that the second address does not exist among the plurality of mapping entries. The switching device further includes: a second port coupled to the routing unit, the second port configured to transmit the first request based on the second address; wherein the routing unit is configured to update the cache with a first mapping entry that associates the first address with the second address obtained from the first device.

[0008] Another embodiment of this disclosure provides a switching device. The switching device includes: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries, the plurality of mapping entries including a first mapping entry associating the first address with a second address; a controller coupled to the cache, the controller configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria; a routing unit coupled to the cache, the routing unit configured to determine the destination of the first request based on the second address; and a second port coupled to the routing unit, the second port configured to transmit the first request to the destination. Attached Figure Description

[0009] A further understanding of the nature and advantages of particular embodiments can be achieved by referring to the remainder of the specification and the accompanying drawings, in which similar reference numerals are used to refer to similar components. In some instances, sublabels are associated with reference numerals to identify one of a plurality of similar components. When reference numerals are used without specifying existing sublabels, it is intended to refer to all such plurality of similar components.

[0010] Figure 1 is a schematic diagram illustrating the architecture of a computing system with an Address Translation Service (ATS) mechanism according to various embodiments of the present invention.

[0011] Figure 2 is a schematic diagram illustrating the architecture of a computing system supporting direct memory access (DMA) according to various embodiments of the present invention.

[0012] Figure 3 is a schematic diagram illustrating a hierarchical PCIe switch system with integrated address translation cache (ATC) according to various embodiments of the present invention.

[0013] Figure 4 is a schematic diagram illustrating various embodiments of a switching device according to the present invention.

[0014] Figure 5 is a schematic diagram illustrating the switch mapping and configuration of a computing system according to various embodiments of the present invention. Detailed Implementation

[0015] This invention relates to a switching device for address translation in a data communication system. In one embodiment, the switching device includes a first port and a cache. The first port is configured to receive a first request associated with a first address, and the cache is configured to store multiple mapping entries. The switching device further includes a routing unit coupled to the cache, configured to determine whether a second address associated with the first address exists in the cache. The cache allows for efficient storage and retrieval of frequently used address translations, thereby reducing the need for repeated access to the host system for address translation. This minimizes latency in processing data requests and improves overall system performance. By retrieving address translations from the cache, the system can optimize data flow and enhance the speed and efficiency of data communication across multiple devices. Other embodiments also exist.

[0016] One general aspect includes a switching device comprising: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries; and a routing unit coupled to the cache. The routing unit is configured to: determine whether a second address, associated with the first address, exists among the plurality of mapping entries; retrieve the second address in response to determining that the second address exists among the plurality of mapping entries; and forward the first request to a first device to obtain the second address in response to determining that the second address does not exist among the plurality of mapping entries, the first device being configured to provide the second address by performing address translation based on the first address. The switching device further includes a second port coupled to the routing unit, the second port being configured to transmit the first request based on the second address. The routing unit is configured to update the cache with the first mapping entry that associates the first address with the second address obtained from the first device.

[0017] The implementation may include one or more of the following features. The switching device further includes a buffer coupled to the routing unit, the buffer being configured to store the first request. The switching device further includes a controller coupled to the cache, the controller being configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria. The predetermined criteria include at least one of access frequency, storage duration, or cache capacity. The first address includes a virtual address. The second address includes a physical address. The first request includes a direct memory access (DMA) request. The first device includes an upstream component. The upstream component includes a second switch or host.

[0018] According to another embodiment, the present invention provides a switching device comprising: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries; and a routing unit coupled to the cache. The routing unit is configured to: determine whether a second address, associated with the first address, exists among the plurality of mapping entries; retrieve the second address in response to determining that the second address exists among the plurality of mapping entries; and forward the first request to a first device to obtain the second address in response to determining that the second address does not exist among the plurality of mapping entries. The switching device further includes a second port coupled to the routing unit, the second port configured to transmit the first request based on the second address. The routing unit is configured to update the cache with the first mapping entry that associates the first address with the second address obtained from the first device.

[0019] The implementation may include one or more of the following features. The switching device further includes a buffer coupled to the routing unit, the buffer being configured to store the first request. The switching device further includes a controller coupled to the cache, the controller being configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria. The predetermined criteria include at least one of access frequency, storage duration, or cache capacity. The first address includes a virtual address. The second address includes a physical address. The first request includes a direct memory access (DMA) request. The first device includes an upstream component.

[0020] According to another embodiment, the present invention provides a switching device comprising: a first port configured to receive a first request associated with a first address; a cache configured to store a plurality of mapping entries, the plurality of mapping entries including a first mapping entry associating the first address with a second address; a controller coupled to the cache, the controller configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria; a routing unit coupled to the cache, the routing unit configured to determine the destination of the first request based on the second address; and a second port coupled to the routing unit, the second port configured to transmit the first request to the destination. In various embodiments, the predetermined criteria include at least one of access frequency, storage duration, or cache capacity. The first address includes a virtual address, and the second address includes a physical address.

[0021] The following description is presented to enable those skilled in the art to make and use the invention and to incorporate it into a particular application context. Various modifications and multiple uses in different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of embodiments. Therefore, the invention is not intended to be limited to the presented embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0022] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without being limited to these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form rather than in detail to avoid obscuring the invention.

[0023] Readers should note all papers and documents submitted with and made publicly available with this specification, the contents of which are incorporated herein by reference. All features disclosed in this specification (including any appended claims, abstracts, and drawings) may be replaced by alternative features for the same, equivalent, or similar purposes, unless expressly stated otherwise. Therefore, unless expressly stated otherwise, each disclosed feature is merely one instance of a range of generally equivalent or similar features.

[0024] Furthermore, any element not expressly referred to in the claims as a “component” for performing the specified function or a “step” for performing a particular function shall not be construed as a “component” or “step” as specified in paragraph 6 of section 112 of 35 USC. Specifically, the use of “step” or “action” in the claims herein is not intended to invoke the provisions of paragraph 6 of 35 USC 112.

[0025] When an element is referred to herein as “connected” or “coupled” to another element, it should be understood that the element may be directly connected to the other element or have an intermediary element present between the elements. In contrast, when an element is referred to as “directly connected” or “directly coupled” to another element, it should be understood that there is no intermediary element in the “direct” connection between the elements. However, the presence of a direct connection does not preclude the possibility of other connections in which intermediary elements may be present.

[0026] Furthermore, the terms left, right, front, back, top, bottom, forward, reverse, clockwise, and counterclockwise are used for interpretive purposes only and are not limited to any fixed direction or orientation. Specifically, they are used merely to indicate the relative position and / or orientation between various parts of an object and / or component.

[0027] Furthermore, for ease of description, the methods and processes described herein may be described in a specific order. However, it should be understood that, unless the context otherwise requires, intermediate processes may occur before and / or after any part of the described process, and various other processes may be reordered, added, and / or omitted according to various embodiments.

[0028] Unless otherwise indicated, all figures used herein to express quantity, size, etc., should be understood to be modified by the term “about” in all instances. In this application, unless specifically stated otherwise, the use of the singular includes the plural, and unless otherwise indicated, the use of the terms “and” and “or” means “and / or”. Furthermore, the use of the terms “including” and “having”, as well as other forms (e.g., “includes”, “included”, “has”, “have”, and “had”), should be considered non-exclusive. Moreover, terms such as “element” or “component” cover both elements and components comprising one unit and elements and components comprising more than one unit, unless specifically stated otherwise.

[0029] As used herein, the phrase “at least one of…” preceding a series of items (where the terms “and” or “or” are used to separate any of the items) modifies the entire list, not each member of the list (i.e., each item). The phrase “at least one of…” does not require the selection of at least one of every listed item; rather, the phrase allows for the inclusion of at least one of any of the items and / or at least one of any combination of the items. For example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; and / or any combination of A, B, and C. In examples where the intention is to select “at least one of each of A, B, and C” or alternatively “at least one of A, at least one of B, and at least one of C”, this is explicitly stated.

[0030] Figure 1 is a schematic diagram illustrating the architecture of a computing system 100 with an Address Translation Service (ATS) mechanism according to various embodiments of the present invention. This figure is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art.

[0031] In various implementations, System 100 represents a distributed computing architecture that interconnects multiple hardware components to facilitate seamless communication and high-speed data transfer. For example, System 100 is designed to support high-speed communication between multiple devices, such as network interface cards (NICs), graphics processing units (GPUs), and storage controllers. These devices are interconnected via switches that facilitate data routing between the devices and external systems, such as host systems and other endpoints. System 100 can be applied in various computing environments, such as data centers, AI / ML workloads, cloud computing, high-performance computing systems, and / or the like.

[0032] In various implementations, PCIe (PCI Fast) is used to facilitate high-speed communication between components. PCIe is a high-speed serial bus interface that allows for low-latency, high-bandwidth data exchange between connected devices such as CPUs, memory, NICs, GPUs, and memory controllers. It supports chip-to-chip and board-to-board interconnects via cards and connectors, allowing multiple devices to communicate through a shared data path. PCIe is particularly useful in high-performance computing environments where large amounts of data need to be efficiently transferred between processing units and memory.

[0033] Depending on the implementation, system 100 may utilize direct memory access (DMA) to transfer data between components. For example, the term "direct memory access" can refer to a process in which devices can directly transfer data between their own memory and system memory without intervention from the CPU. This mechanism reduces CPU overhead and accelerates data transfer rates, which is beneficial in high-performance computing environments where multiple devices frequently exchange large amounts of data. For example, in AI / ML workloads, the NIC can directly transfer data to the GPU for processing without requiring the CPU to handle each transaction.

[0034] According to some embodiments, system 100 may include device 101. For example, device 101 may include endpoint devices. The term "endpoint" or "endpoint device" may refer to any device connected to a shared bus that communicates with other components in the system via a switch or root complex. Examples of endpoints may include, but are not limited to, NICs, GPUs, storage devices, and / or other peripheral devices. In some instances, device 101 may perform DMA and communicate directly with another endpoint device. For example, device 101 may include a NIC that can bypass system memory and directly transfer large datasets to the GPU for processing.

[0035] In various implementations, for a DMA operation to occur, the endpoint device (e.g., device 101) needs to perform address translation to ensure it is communicating using the correct memory address. For example, the term "address translation" can refer to the process of translating one type of memory address to another. This could involve translating a virtual address (VA) used by the device to a physical address (PA) used by the system memory, or vice versa. Address translation ensures that memory access requests from the endpoint device are correctly routed to the appropriate location in physical memory. In some instances, device 101 may operate using virtual addresses, which represent abstract memory locations within a virtual memory space assigned to the device. Virtual addresses can be mapped by the system to physical addresses to enable actual data transfers. For example, the term "physical address" refers to the actual location of data in the system memory that the hardware uses to directly access memory.

[0036] In some embodiments, device 101 interacts with host system 110 to perform address translation. For example, the term "host" or "host system" may refer to a central component that manages and coordinates the operation of connected devices. Host system 110 may be responsible for managing address translation between devices and coordinating communication between devices. In various instances, host system 110 may include at least one of a memory 103, a memory controller 104, a processor 109, a memory management unit (MMU) 111, an address translation and protection table (ATPT) 106, a root complex 105, a first cache 107, an input / output memory management unit (IOMMU) 108, and / or the like.

[0037] In various implementations, device 101 relies on an Address Translation Service (ATS) to perform address translation. An ATS can be defined by the PCIe standard as a mechanism that allows PCIe devices to request and manage their address translation. When device 101 needs to access memory, it generates an ATS request 113 and sends it to root complex 105. For example, the term "root complex" can refer to a component in a system hierarchy that connects a host system to an endpoint. Root complex 105 acts as a bridge between device 101 and host system 110 by forwarding memory access requests (e.g., ATS request 113) and ensuring proper communication between the endpoint device and host system 110.

[0038] In some instances, root complex 105 forwards ATS requests to IOMMU 108 for translation. IOMMU 108 handles address translation for input / output (I / O) devices (e.g., device 101) and ensures that each device accesses only its authorized memory regions. In some embodiments, address translation may also involve processor 109 and MMU 111 when processor 109 needs memory access. For example, the term "processor" may refer to a central processing unit or other computing unit responsible for executing instructions and managing the overall operation of the system. In some cases, processor 109 may generate virtual addresses when performing memory operations, which may be translated into physical addresses by MMU 111. Instances of memory management units may include, but are not limited to, CPUMMU, GPU MMU, virtual MMU, and / or the like. Depending on the implementation, MMU 111 may be implemented as a separate dedicated hardware unit or directly integrated within the CPU as part of a system-on-a-chip (SoC) architecture.

[0039] In various implementations, ATPT 106 is configured to store a mapping of virtual addresses to their corresponding physical addresses. During the translation process, IOMMU 108 or MMU 111 can refer to ATPT 106 to retrieve the appropriate physical address based on the virtual address provided by device 101 or processor 109. For example, if device 101 needs to transfer data to memory 103, IOMMU 108 can access ATPT 106 to translate the address and complete the data transfer. ATPT 106 ensures that memory access is safe and efficient by maintaining the up-to-date mapping of memory addresses in the system. Once the address translation is complete, root complex 105 can send an ATS response 114 back to device 101, providing the translated physical address. Device 101 can then use this physical address to perform the required DMA operation and access the correct location in memory 103.

[0040] In some embodiments, memory controller 104 may be configured to manage communication between processor 109 and memory 103, ensuring that data requests from a device (e.g., device 101) are processed efficiently. For example, the term "memory controller" may refer to a hardware component that manages data flows to and from memory. Depending on the implementation, memory controller 104 may comprise an integrated controller within a SoC or a dedicated controller within a memory management subsystem. In some cases, once a virtual address from device 101 is translated to a physical address, memory controller 104 coordinates the transfer of data to or from the location in memory 103. Memory 103 may comprise the system's primary storage device (e.g., random access memory (RAM) or other types of volatile / non-volatile memory) where data is temporarily or permanently stored. Memory 103 may be used to store program instructions, operational data, system configuration, and / or the like.

[0041] To optimize address translation efficiency, host system 110 may also include a first cache 107. For example, the term "cache" can refer to a memory or storage component that temporarily stores frequently accessed data, instructions, or address maps to reduce the time required to retrieve them. In some instances, the first cache 107 may include an address translation cache (ATC), which can be configured to store recently translated address maps. This allows host system 110 to quickly retrieve the necessary maps for future translation requests, rather than performing translation from scratch.

[0042] To reduce the overhead of address translation performed by host system 110, address translation can also be performed locally by device 101. In various embodiments, device 101 may include a second cache 102 that stores recently translated address maps at the device level. By caching these translations, device 101 can avoid sending repeated ATS requests for the same memory region, thereby reducing the overhead associated with frequent address translation. If the required address map is available in ATC 102, device 101 can retrieve the physical address directly from the cache without requesting a translation from host system 110, thus speeding up memory access operations.

[0043] However, the capacity of the second cache 102 may be limited, meaning it can only store a limited number of address maps at a time. This limitation can lead to cache misses, requiring device 101 to rely on host system 110 for address translation. This could result in device 101 needing to resend ATS requests for addresses previously cached but removed due to space constraints. Frequent cache misses can increase latency, especially in systems with high data throughput or workloads involving frequent access to large sets of memory areas. In such cases, optimizing cache management across multiple levels of the system becomes beneficial for minimizing latency and improving maintenance performance.

[0044] Figure 2 is a schematic diagram illustrating the architecture of a computing system 200 supporting direct memory access (DMA) according to various embodiments of the present invention. This figure is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art.

[0045] In various implementations, computing system 200 includes multiple devices connected via switch 203 to facilitate high-speed communication and DMA operations. For example, system 200 may include one or more endpoint devices, such as NIC 201 and GPU 202. NIC 201 can handle network communication and data transfer to and from external networks. In systems where large amounts of data need to be ingested or distributed, such as in cloud computing or high-performance data centers, NICs facilitate efficient data movement across systems. In some instances, GPU 202 can be used to handle computationally intensive tasks, such as AI model training, parallel data processing, or high-speed rendering.

[0046] In some embodiments, one or more endpoint devices (e.g., NIC 201 and GPU 202) may be coupled to switch 203, which facilitates data routing between the endpoint devices and host system 208. For example, the term "switch" may refer to a hardware component that facilitates communication between multiple devices by managing data flow across a shared communication path. Examples of switches may include, but are not limited to, PCIe switches, Ethernet switches, wireless bandwidth switches, Fibre Channel switches, and / or the like. In some instances, switch 203 includes a PCIe switch designed to connect various PCIe-compatible devices, such as NICs, GPUs, storage devices, and other peripherals. The PCIe switch acts as an intermediary layer between these devices and host system 208, facilitating high-speed data transfer between devices on the PCIe bus.

[0047] According to various embodiments, host system 208 may include multiple components for managing memory access and address translation. For example, host system 208 may include at least one of memory 207, CPU 206, root complex 204, IOMMU 205, and / or the like. Root complex 204 may act as an intermediary layer between endpoint devices (e.g., NIC 201 and GPU 202) and CPU 206, controlling data flow and ensuring communication between the devices and system memory 207. Memory 207 may be used to store data for processing and communication between devices. Depending on the implementation, memory 207 may be any suitable type of volatile or non-volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, or other memory storage systems. IOMMU 205 may be responsible for performing address translation between virtual addresses used by endpoint devices and corresponding physical addresses in memory 207, thereby ensuring secure and efficient data transfer during DMA operations.

[0048] In some implementations, switch 203 may include an address translation cache to facilitate address translation. The ATC can store recently translated address maps to accelerate the process of retrieving physical addresses without constantly returning to the reference host system 208. This local cache allows switch 203 to quickly access maps when accessing the same memory area multiple times, thereby reducing latency and improving overall system efficiency.

[0049] In some implementations, switch 203 may be configured to monitor (or “spy”) ATS transactions between endpoint devices and root complex 204. When an endpoint device (e.g., NIC 201 or GPU 202) sends an ATS request to root complex 204 to perform address translation, switch 203 can observe the transaction and add the translated address to its own ATC. This caching process reduces the need for subsequent ATS requests for the same address and can enhance performance by minimizing the frequency of address translation required by host system 208. Additionally, switch 203 helps alleviate the burden on endpoint devices with limited local cache capacity by providing an additional caching layer, allowing endpoint devices to retrieve translated addresses directly from the switch’s cache.

[0050] Figure 3 is a schematic diagram illustrating a hierarchical PCIe switch system 300 with integrated Address Translation Cache (ATC) according to various embodiments of the present invention. This figure is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be appreciated by those skilled in the art.

[0051] In various implementations, system 300 includes multiple switches (e.g., switches 304a, 304b, 304c, and 304d) that facilitate communication between various endpoint devices (e.g., NICs 301a to 301c, GPUs 303a to 303c, and / or other endpoint devices 305a to 305d) and the host system. The switches can be connected in a hierarchical structure, allowing for efficient data routing across multiple devices. In some instances, the host system may include at least one of a root complex 306, an IOMMU 307, a CPU 308, a memory 309, and / or the like.

[0052] According to some embodiments, one or more endpoint devices (e.g., NICs and GPUs) may implement an ATS and maintain their own ATC (e.g., ATC 302a to e), allowing them to perform local address translation on a per-transaction basis without relying on the host system. However, the number of cache entries in an endpoint device may be limited. In some cases, one or more switches may include an ATC that stores recently translated address mappings to reduce latency and improve data transfer performance. For example, switches 304a to 304c may include ATCs (e.g., ATC 302f, ATC 302g, and ATC 302h) that help store and manage address mappings for various endpoint devices. For example, endpoint devices such as NICs 301a to b may rely on their own ATCs (e.g., ATC 302a to b) to store address translations. When these ATCs run out of storage capacity, the switch can provide additional cache support through its integrated ATCs.

[0053] In some implementations, when an endpoint device (e.g., NIC 301a) initiates a DMA transaction, it sends a request containing a virtual address. If the endpoint's local ATC contains the corresponding physical address, a cache hit occurs (e.g., indicated by a solid line), allowing the DMA operation to proceed without further delay. If the required translation is not found in the local ATC, the request can be forwarded to a higher-level component in the hierarchy (e.g., switch 304a). If switch 304a's ATC contains the necessary translation, the transaction can proceed immediately. If switch 304a's ATC does not contain the necessary translation, the request can then be forwarded to the next switch in the hierarchy (e.g., switch 304b, switch 304c, etc.), which can also check its own ATC for address mapping. In this hierarchical structure, each switch in the chain has the opportunity to handle address translation, thereby reducing the need for the request to reach the host system. This hierarchical approach improves overall efficiency by reducing the load on the host system, thus accelerating memory access operations.

[0054] In some instances, if neither the endpoint device nor the PCIe switch has the required translation, a cache miss continues (e.g., indicated by a dashed line), and the request can be escalated to the host system. For example, if the NIC 301c or GPU 303d encounters a cache miss in its local ATC, and the ATC of the switch 304c (e.g., ATC 302h) also does not contain the required mapping, the request can be forwarded to the root complex 306 in the host system. From there, the IOMMU 307 performs address translation, retrieves the correct physical address, and sends the translated address back up the hierarchy. This multi-level caching approach reduces the need for frequent address translation requests at higher levels, thereby ensuring that most transactions are processed locally within the PCIe hierarchy.

[0055] In various implementations, system 300 may implement a hierarchical caching system that uses a portion of host memory 309 as an extended cache (e.g., ATC 302i, 302j) to store extra cache entries when the ATC in a switch or endpoint device runs out of space. When a switch (e.g., switches 304b to c) cannot store all the necessary mappings in its own ATC due to space constraints, it can retrieve entries (e.g., indicated by dashed lines) from the extended cache maintained in system memory 309. This extended cache in memory 309 provides additional storage for address mapping, thereby ensuring that the system can continue to perform address translation efficiently even when the local hardware cache in the PCIe switch or endpoint device is full.

[0056] Figure 4 is a schematic diagram illustrating various embodiments of a switching device 400 according to the present invention. This figure is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art.

[0057] In various implementations, switch device 400 may be part of a larger distributed system (e.g., system 300 of Figure 3). Switch 400 may be configured to manage data routing, memory access requests, and address translation between multiple endpoint devices (e.g., NICs, processors, or other peripherals) and external networks, thereby ensuring seamless communication and high-speed data transfer. It can be deployed in a variety of applications, such as data centers, cloud computing infrastructure, high-performance computing systems, AI training environments, ML processing, and / or the like. Depending on the implementation, switch device 400 may be integrated into a hierarchical system involving multiple layers of switches and devices, and may utilize distributed caching mechanisms (e.g., ATC) to optimize memory access and reduce latency across system hierarchies.

[0058] As shown, switch device 400 may include one or more ports (e.g., first port 401a and second port 401b). For example, the term "port" may refer to a physical or logical interface on the switch through which data can be transmitted and received. Ports serve as connection points between endpoint devices (e.g., NICs, processors) and external networks, thereby allowing data to flow between these components. Examples of ports may include, but are not limited to, PCIe ports, Ethernet ports, wireless bandwidth ports, or other communication interfaces. Depending on the implementation, ports may be used as upstream ports or downstream ports. Upstream ports connect the switch to upstream components (e.g., host systems or higher-level networks), while downstream ports connect the switch to downstream components (e.g., endpoint devices).

[0059] In various implementations, the switch device 400 may be implemented as a PCIe switch and may be coupled to one or more endpoint devices (e.g., NIC, GPU, storage controller, etc.). The one or more endpoint devices may be connected via a PCIe interface. For example, the term "PCIe interface" may refer to a physical or logical connection that allows devices to communicate via the PCIe standard.

[0060] In some embodiments, the first port 401a may be configured to receive a first request associated with a first address. For example, the term "request" may refer to a communication or command sent by a device to initiate a specific operation (e.g., data retrieval, memory access, or processing task). An endpoint device (such as a NIC, GPU, or other peripheral device) may generate a request when it needs to perform a task such as DMA or data transfer. Instances of requests may include, but are not limited to, memory read requests, write requests, address translation requests, and / or the like. The term "address" may refer to a location in memory or a communication destination used by the device to access data or transfer information. Depending on the implementation, the first address may include a virtual address or a physical address.

[0061] In some embodiments, the switching device 400 further includes one or more processing layers responsible for various stages of data processing, error detection, and protocol management as data flows through the switch. The one or more processing layers may include, but are not limited to, SerDes layers 402a to b, physical layers 403a to b, multiplexer / demultiplexer layers 404a to b, data link layers 405a to b, transaction layers 406a to b, and / or similar layers.

[0062] In some implementations, SerDes layers 402a to 402b may include serializer-deserializer circuitry that converts parallel data into serial data for transmission over high-speed communication links, and then converts the serial data back into parallel data for further processing. SerDes layers 402a to 402b achieve high-speed data transmission by reducing the number of data lines required for communication, which helps maintain high data transfer rates between devices.

[0063] After the SerDes conversion, data can move through physical layers 403a to 403b, which are responsible for handling the physical transmission of data across the communication medium, ensuring that signals are properly synchronized and transmitted with minimal loss. Multiplexer / demultiplexer layers 404a to 404b manage the data stream by combining multiple data signals into a single stream (e.g., multiplexing) or separating a single data stream into multiple signals (e.g., demultiplexing). These processing layers achieve efficient use of the communication channel by dynamically managing available bandwidth and ensuring that data is transmitted to the appropriate endpoints.

[0064] In various embodiments, data link layers 405a to 405b and transaction layers 406a to 406b handle higher-level communication protocols, ensuring that data packets are properly formatted, validated, and transmitted across the switch device 400. For example, data link layers 405a to 406b provide error detection and correction mechanisms to ensure that data transmitted between devices is reliable and error-free. Transaction layers 406a to 406b manage the actual data transmission transactions between devices, determining how data is sent, received, and processed at each endpoint.

[0065] According to some embodiments, the switch device 400 may include a switch core 413. For example, the term "switch core" refers to the central processing unit of the switch that manages the overall data flow and controls how data is routed and processed within the switch. The switch core 413 may be configured to control the internal operation of the switch device 400, manage how data flows between ports, and coordinate communication between connected devices. In various instances, the switch core 413 may facilitate address translation by managing how requests for address translation are processed and directing these requests to flow between different components within the switch device 400. For example, the switch core 413 may include at least one of a buffer 407, a routing unit 408, an arbitration unit 409, a scheduler 410, a controller 411, a cache 412, and / or the like.

[0066] In some instances, switch core 413 further includes a buffer 407, which can be configured to store a first request. For example, the term "buffer" can refer to a memory element or storage area used to temporarily store data. Buffer 407 is used to smooth the data flow by accommodating differences in data transfer rates between different components or devices. In some cases, data arriving from the NIC or external network may arrive at a rate higher than the system can handle; buffer 407 can temporarily store this data until the system is ready to process the data or transmit it to its final destination. By storing data before it is processed, buffer 407 ensures that the system can handle multiple requests simultaneously, preventing bottlenecks and maintaining a stable flow of information through switch device 400.

[0067] In various implementations, switch core 413 further includes cache 412. For example, cache 412 may include an ATC (Automatic Address Translation) that can be configured to store multiple mapping entries. For example, the term "mapping entry" may refer to a record that associates a first address (e.g., a virtual address) with a second address (e.g., a physical address). Mapping entries can be used during address translation processes, where a virtual address used by an endpoint device must be translated into a physical address corresponding to a specific location in memory for data transfer operations (e.g., DMA). In some instances, multiple mapping entries may contain additional metadata, such as the access frequency of each entry, the time the entry was added to the cache, or the size of the memory area associated with the address.

[0068] In some embodiments, switch core 413 further includes routing unit 408. For example, the term "routing unit" may refer to a component responsible for determining the path data takes within the switch, thereby ensuring that the data is directed to the appropriate device or network destination. Routing unit 408 is responsible for determining the appropriate destination for a request based on the address information contained in each request. Where address translation is required, routing unit 408 coordinates with other components (e.g., cache 412) to facilitate efficient memory access operations.

[0069] In various instances, routing unit 408 may be coupled to cache 412. When a first request is received from an endpoint device at port 401a, routing unit 408 is responsible for determining whether the corresponding address mapping exists in the cache. In some instances, routing unit 408 queries cache 412 to determine whether a second address exists among multiple mapping entries. The second address may be associated with the first address. Depending on the implementation, the second address may contain a virtual address or a physical address. For example, the first address may contain a virtual address used by the endpoint device (e.g., NIC or GPU) to access memory. This virtual address does not directly correspond to a physical memory location and must be translated into a second address containing the corresponding physical address where the data resides in memory.

[0070] Routing unit 408 plays a crucial role in the address translation process by determining whether a second address (e.g., a physical address) is already available in cache 412, which stores previously translated address mappings. For example, in response to determining that a second address exists among multiple mapping entries (e.g., a cache hit), routing unit 408 can retrieve the second address from cache 412 and determine the appropriate destination for the first request based on the second address. By retrieving the second address from cache 412, routing unit 408 avoids the need to request address translation from upstream components (e.g., the host system), thereby reducing latency and speeding up the overall data transmission process.

[0071] In response to determining that a second address is not present in multiple mapping entries (e.g., cache miss), routing unit 408 may perform address translation by forwarding the first request to an upstream component (e.g., a host system or higher-level network). In some instances, routing unit 408 may forward the first request to a first device to obtain a second address. The first device may be configured to provide the second address by performing address translation based on the first address. Once address translation is complete, the second address may be returned to switching device 400. Routing unit 408 may determine an appropriate destination based on the second address and route the first request to the destination via a second port 401b. In various instances, routing unit 408 may also update cache 412 with a new mapping entry (e.g., a first mapping entry) that associates the first address with the second address, thereby ensuring that future requests involving the same virtual address can be handled more efficiently.

[0072] In some implementations, the switch core 413 further includes a controller 411 that may be coupled to the cache 412. For example, the term "controller" may refer to a hardware or software component responsible for managing the operation of one or more elements within the device. Depending on the implementation, the controller 411 may be implemented as dedicated hardware circuitry, a programmable logic unit, or an embedded software module.

[0073] In some embodiments, controller 411 is configured to manage the operation of cache 412 and regulate how mapping entries are stored and maintained within cache 412. For example, controller 411 may be responsible for implementing cache management strategies based on predefined criteria such as access frequency, storage duration, cache capacity, and / or the like.

[0074] As an example, the term "access frequency" can refer to how often a system requests or uses a particular mapping entry. Frequently accessed mapping entries are more likely to remain associated with future address translation requests. Controller 411 can track the frequency of access to each entry in cache 412 and prioritize the retention of frequently accessed mappings. For example, mappings regularly accessed by high-performance computing tasks or AI training models can be kept in cache 412 for a longer period to avoid duplicate address translation requests from external components. Conversely, less frequently accessed mappings can be de-prioritized and eventually removed from cache 412 to make room for more critical entries.

[0075] In some instances, the term "storage duration" may refer to the length of time a mapped entry has been stored in cache 412. For example, controller 411 may apply policies to limit the retention time of specific entries, especially if they have not been accessed for a period of time. For instance, if the first mapped entry is not used within a predefined period, the controller may decide to remove it to free up space for a new entry. This approach helps ensure that stale or obsolete mappings do not occupy valuable space in cache 412, thereby maintaining cache efficiency and ensuring that only useful, up-to-date mappings are stored.

[0076] In some cases, the term "cache capacity" may refer to the total amount of space available in cache 412 for storing mapped entries. Since the size of the cache memory may be limited, controller 411 can adjust how many entries can be stored at any given time. If the cache capacity is full and new entries need to be added (e.g., after a cache miss), controller 411 can decide which existing entries should be replaced or evicted. For example, controller 411 can remove least recently used (LRU) or least frequently accessed entries to maintain an optimal cache size.

[0077] In various implementations, scheduler 410 may be coupled to controller 411. For example, the term "scheduler" may refer to a component responsible for the timing and coordination of tasks within the system. Instances of schedulers may include, but are not limited to, round-robin schedulers, priority-based schedulers, credit-based schedulers, and / or similar types. Scheduler 410 may be configured to manage the execution and sequencing of data transmission tasks, thereby ensuring that resources are allocated efficiently and devices operate synchronously. Depending on the implementation, scheduler 410 may be configured to coordinate data flows, manage task timing, and / or detect the operational status of endpoint devices.

[0078] In some embodiments, the switch core 413 further includes an arbitration unit 409. For example, the term "arbitration unit" may refer to a component responsible for managing access to shared resources, such as data paths or communication channels. In various instances, when multiple devices connected to the switch device 400 simultaneously request access to the same resource, the arbitration unit 409 determines which device receives priority based on predefined rules or scheduling algorithms. This process ensures efficient data flow between devices and prevents resource contention or service bottlenecks. Examples of arbitration mechanisms include priority-based arbitration, round-robin arbitration, and weighted fair queuing.

[0079] Figure 5 is a schematic diagram illustrating the switch mapping and configuration of a computing system 500 according to various embodiments of the present invention. This figure is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art.

[0080] In various implementations, system 500 includes multiple endpoint devices 501a to c, switches 503a to b, and a root complex 505 interconnected via data links, thus representing a distributed switching architecture. Endpoint devices (e.g., NICs, GPUs, or storage controllers) are responsible for initiating and processing data transfers across the network. For example, endpoint device 501a may communicate with switch 503a via link 502a. Endpoint devices 501b and 501c may communicate with switch 503b via links 502b and 502c, respectively. Each endpoint device may send data or requests through a switch that manages data routing and address translation as part of the communication process.

[0081] In some embodiments, each link in system 500 may be configured to support various virtual channels (VCs) (e.g., VC0, VC1, VC2, and VC3), which manage service categories (e.g., TC[0:1], TC[2:4], TC[5:6], TC7) that prioritize different types of data streams. These service categories represent the priority of data packets, allowing critical data to be transmitted with higher priority than lower priority packets. The mapping between service categories and virtual channels ensures efficient bandwidth allocation and appropriate data flow control within the switching infrastructure. Depending on the application, switches 503a and 503b may support multiple layers of virtual channels and service categories to handle high-priority data, bulk transmissions, time-sensitive operations, and / or the like.

[0082] Root complex 505 can be configured to connect switches (e.g., switches 503a to b) to a wider system (e.g., host systems), manage how data is routed between various endpoint devices, and coordinate communication across the network. For example, switches 503a to b can be coupled to root complex 505 via links 504a to b, respectively. In some embodiments, root complex 505 interacts with multiple switches (e.g., 503a and 503b) to optimize traffic flow between endpoint devices and higher-level system components.

[0083] In various implementations, switch 503a includes cache 506a, and switch 503b includes cache 506b. These caches are responsible for storing address translation entries that associate virtual addresses with physical addresses. When an endpoint device (e.g., 501a or 501b) initiates a request for data transfer, the switch (e.g., 503a or 503b) can check its caches to determine if the required address translation has already been stored. If the cache contains the mapping, the switch can directly route the request to the correct memory location, thereby reducing the time required for the translation process.

[0084] It should be understood that caches 506a and 506b in the switch help alleviate the burden on endpoint devices, which may have limited cache capacity. When the local cache of an endpoint device (e.g., 501a or 501b) runs out of capacity, the switch's cache can act as an additional storage layer, thereby further improving the system's ability to efficiently handle large volumes of data transfers. In the event of a cache miss in either the endpoint device or the switch, the request can be forwarded to the root complex 505 or the host system's IOMMU for address translation. Once translation is complete, the result can be stored in the endpoint's local cache and / or the switch's cache to reduce latency for future requests involving the same address.

[0085] While the foregoing is a complete description of specific embodiments, various modifications, alternative constructions, and equivalents may be used. Therefore, the above description and illustrations should not be construed as limiting the scope of the invention as defined by the appended claims.

Claims

1. A switching device, comprising: A first port, configured to receive a first request associated with a first address; A cache that is configured to store multiple mapping entries; A routing unit coupled to the cache, the routing unit being configured to: determine whether a second address exists among the plurality of mapping entries, the second address being associated with the first address; and, in response to determining that the second address exists among the plurality of mapping entries, retrieve the second address; In response to determining that the second address does not exist in any of the plurality of mapping entries, the first request is forwarded to a first device to obtain the second address, the first device being configured to provide the second address by performing address translation based on the first address; and a second port, which is coupled to the routing unit, the second port being configured to transmit the first request based on the second address; The routing unit is configured to update the cache with a first mapping entry that associates the first address with the second address obtained from the first device.

2. The device of claim 1, further comprising a buffer coupled to the routing unit, the buffer being configured to store the first request.

3. The device of claim 1, further comprising a controller coupled to the cache, the controller being configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria.

4. The device of claim 3, wherein the predetermined criteria include at least one of access frequency, storage duration, or cache capacity.

5. The device according to claim 1, wherein the first address includes a virtual address.

6. The device according to claim 1, wherein the second address includes a physical address.

7. The device of claim 1, wherein the first request includes a direct memory access (DMA) request.

8. The device of claim 1, wherein the first device includes an upstream component.

9. The device of claim 8, wherein the upstream component includes a second switch or a host.

10. A switching device, comprising: A first port, configured to receive a first request associated with a first address; A cache that is configured to store multiple mapping entries; A routing unit coupled to the cache, the routing unit being configured to: determine whether a second address exists among the plurality of mapping entries, the second address being associated with the first address; and, in response to determining that the second address exists among the plurality of mapping entries, retrieve the second address; And in response to determining that the second address does not exist in the plurality of mapping entries, the first request is forwarded to the first device to obtain the second address; and a second port, which is coupled to the routing unit, the second port being configured to transmit the first request based on the second address; The routing unit is configured to update the cache with a first mapping entry that associates the first address with the second address obtained from the first device.

11. The device of claim 10, further comprising a buffer coupled to the routing unit, the buffer being configured to store the first request.

12. The device of claim 10, further comprising a controller coupled to the cache, the controller being configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria.

13. The device of claim 12, wherein the predetermined criteria include at least one of access frequency, storage duration, or cache capacity.

14. The device of claim 10, wherein the first address includes a virtual address.

15. The device of claim 10, wherein the second address includes a physical address.

16. The device of claim 10, wherein the first request includes a direct memory access (DMA) request.

17. The device of claim 10, wherein the first device includes an upstream component.

18. A switching device, comprising: A first port, configured to receive a first request associated with a first address; A cache configured to store multiple mapping entries, the multiple mapping entries including a first mapping entry that associates the first address with the second address; A controller coupled to the cache, the controller being configured to manage the plurality of mapping entries stored in the cache based on predetermined criteria; A routing unit coupled to the cache, the routing unit being configured to determine the destination of the first request based on the second address; and a second port coupled to the routing unit, the second port being configured to transmit the first request to the destination.

19. The device of claim 18, wherein the predetermined criteria include at least one of access frequency, storage duration, or cache capacity.

20. The device of claim 18, wherein the first address includes a virtual address and the second address includes a physical address.

Images