A multi-host system and a cross-domain access method

Abstract

This application provides a multi-host system and a cross-domain access method. The system includes: a structure manager and at least two host domains; the structure manager is used to discover and manage the topology of the multi-host system, assign segment numbers to hosts, and assign port identification numbers to host edge switches, edge ports, and downstream edge switches; the structure manager obtains device information of terminal devices and sends it to the hosts; the hosts send request messages to the terminal devices through the host edge switches via a port-based routing and switching network; the downstream edge switches establish an association between the segment numbers and port identification numbers carried in the request messages, and simultaneously map the physical address of the host requiring cross-domain access to the physical address of the host being accessed; the downstream edge switches receive completion messages, determine the port identification number corresponding to the segment number carried in the completion message based on the association, and then carry the port identification number as a destination identifier in the completion message and send it to the corresponding host.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of chips, and particularly relates to a multi-host system and a cross-domain access method. BACKGROUND

[0002] PCIe is a high-speed serial computer expansion bus standard, which mainly provides a bus interface for CPU to access peripheral devices. The PCIe topology is a tree topology, and each host manages a respective tree topology. Each host controls its own downstream port to form an independent address domain, and the addresses between hosts are completely isolated and cannot access each other. With the continuous development of technology, a single host cannot meet the application of the system, and the application demand of multi-hosts is increasing. Multi-hosts allow multiple computing nodes (hosts) to share and access one or more common hardware resources. In this way, the resource utilization is improved, the flexibility and scalability of the system are enhanced, and the load management and fault recovery strategy are supported. Realizing cross-domain access of multi-hosts and multi-devices has become a research topic. SUMMARY

[0003] The application aims to provide a multi-host system and a cross-domain access method, and aims to solve the technical problems of low performance of cross-domain access terminal devices and dependence on tree topology structure in the multi-host system.

[0004] According to a first aspect of the application, a multi-host system is provided, comprising: a structure manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device;

[0005] The structure manager is configured to discover and manage the topology structure of the multi-host system, and assign a segment number to the host in the host domain and assign a port identification number to the host edge switch, the downstream edge switch and the terminal device based on the topology structure; the structure manager is further configured to obtain device information of the terminal device, and send the device information to the host which needs to access the terminal device across domains;

[0006] In the cross-domain access process, the host which needs to access the terminal device across domains sends a request message to the terminal device which is accessed across domains via a port-based routing switch network through the host edge switch;

[0007] After receiving the request message, the downstream edge switch associated with the terminal device being accessed across domains establishes an association between the segment sequence number and port identifier carried in the request message, maps the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and forwards it to the terminal device being accessed across domains. The downstream edge switch associated with the terminal device being accessed across domains also receives a completion message returned by the terminal device being accessed across domains, and determines the port identifier corresponding to the segment sequence number carried in the completion message based on the association, and carries the port identifier corresponding to the segment sequence number carried in the completion message as the destination identifier in the completion message, which is then sent to the corresponding host by the port-based routing and switching network.

[0008] In an optional implementation, the device information includes the DPA space size of the terminal device: the structure manager uses a custom command to obtain the DPA space size of the terminal device through the component command interface of the downstream edge switch;

[0009] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the terminal device's DPA and latches the base address of the cross-domain host physical address of the terminal device's DPA carried in the configuration message.

[0010] The edge port of the downstream edge switch maps the physical address of the host to be accessed across domains carried in the request message to the physical address of the host to be accessed across domains of the terminal device based on the base address.

[0011] In an optional implementation, the host requiring cross-domain access is further configured to configure a routing table for the host edge switch requiring cross-domain access. The routing table includes a first mapping relationship between the physical address of the host requiring cross-domain access and the port identifier of the edge port connected to the terminal device that can be accessed across domains, and a second mapping relationship between the port identifier and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0012] In an optional implementation, the first mapping relationship is the mapping relationship between the high-order address of the physical address of the host requiring cross-domain access and the port identification number; the low-order address of the physical address of the host requiring cross-domain access is the offset address of the terminal device corresponding to the port identification number.

[0013] In an optional implementation, the downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host to be accessed across domains carried in the request message with the base address of the physical address of the host to be accessed across domains of the terminal device being accessed across domains, thereby realizing the address mapping from the physical address of the host to the physical address of the host to be accessed across domains.

[0014] In an optional implementation, the request message carries the physical address of the host to be accessed across domains and the port identifier of the terminal device to be accessed across domains; after receiving the request message from the host, the host edge switch determines the edge port through the routing table and transmits the request message from the edge port to the port-based routing and switching network.

[0015] In an optional implementation, when the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host being accessed across domains.

[0016] According to a second aspect of this application, a cross-domain access method is provided, implemented in a multi-host system, the multi-host system comprising: a structure manager and at least two host domains; each host domain comprising a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the structure manager is used to discover and manage the topology of the multi-host system, and assign segment sequence numbers to hosts in the host domains and assign port identifiers to the host edge switch, downstream edge switch, and terminal device based on the topology; the structure manager is further used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains; the cross-domain access method includes:

[0017] During cross-domain access, the host requiring cross-domain access sends a request message to the terminal device being accessed via the associated host edge switch through a port-based routing and switching network.

[0018] After receiving the request message, the downstream edge switch associated with the terminal device being accessed across domains establishes the association between the segment sequence number and port identifier carried in the request message, maps the physical address of the host to be accessed across domains in the request message to the physical address of the host being accessed across domains, and then forwards it to the terminal device being accessed across domains.

[0019] When the downstream edge switch associated with the terminal device being accessed across domains receives the completion message returned by the terminal device being accessed across domains, and determines the port identifier corresponding to the segment sequence number carried in the completion message based on the association relationship, the port identifier corresponding to the segment sequence number carried in the completion message is carried as the destination identifier in the completion message and sent to the corresponding host by the port-based routing and switching network.

[0020] In an optional implementation, the device information includes the DPA space size of the terminal device; the method further includes:

[0021] The structure manager sends a custom command to the component command interface of the downstream edge switch to obtain the DPA space size of the terminal device;

[0022] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the terminal device's DPA and latches the base address of the physical address of the cross-domain host carrying the DPA of the terminal device in the configuration message.

[0023] The step of mapping the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwarding it to the terminal device being accessed across domains, includes:

[0024] The edge port of the downstream edge switch maps the physical address of the host to be accessed across domains carried in the request message to the physical address of the host to be accessed across domains of the terminal device based on the base address.

[0025] In an optional implementation, the method further includes:

[0026] The host requiring cross-domain access is also configured to configure a routing table for the host edge switch requiring cross-domain access. The routing table includes a first mapping relationship between the physical address of the host requiring cross-domain access and the port identifier of the terminal device that can be accessed across domains, and a second mapping relationship between the port identifier and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0027] In an optional implementation, the first mapping relationship is the mapping relationship between the high-order address of the physical address of the host requiring cross-domain access and the port identification number; the low-order address of the physical address of the host requiring cross-domain access is the offset address of the terminal device corresponding to the port identification number.

[0028] In an optional implementation, the step of mapping the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwarding it to the terminal device being accessed across domains, includes:

[0029] The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host to be accessed across domains carried in the request message with the base address of the physical address of the host to be accessed across domains of the terminal device being accessed across domains, thereby realizing the address mapping from the physical address of the host to the physical address of the host to be accessed across domains.

[0030] In an optional implementation, the request message carries the physical address of the host requiring cross-domain access and the port identifier of the edge port connecting the terminal device being accessed across domains; the method further includes:

[0031] After receiving the request message from the host, the host edge switch determines the edge port through the routing table and transmits the request message from the edge port to the port-based routing and switching network.

[0032] In an optional implementation, the method further includes:

[0033] When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host being accessed across domains.

[0034] Compared with related technologies, the technical solution of this application has the following advantages:

[0035] This application uses FM to uniformly manage the entire topology and obtains the DPA space size of cross-domain devices based on user-defined commands, preparing for the host to establish a GIM (Global Integrated Memory) address space. PBR routing is used on cross-domain paths, freeing cross-domain communication from being limited to the tree-like topology of PCIe, enabling cross-domain communication in more complex topologies.

[0036] On the cross-domain device side, the DSP above the device implements the segment / PID mapping table, cross-domain address mapping conversion logic, and UIO request message processing logic. This processing logic is used to extract the DPID information from the UIO request message (the purpose is to convert the PBR packet into an HBR packet) and replace the HPA. This processing logic is simple to implement and does not introduce significant overhead in terms of resources and area.

[0037] Based on the CXL protocol and following the PBR routing method, this application allows cross-domain paths and non-cross-domain paths to share physical ports, meaning they can share physical paths without needing to allocate dedicated physical ports for cross-domain paths.

[0038] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures and processes shown in the description and the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram of the physical topology of a cross-domain example of a multi-host system provided in this application is shown.

[0041] Figure 2 A topology diagram from a logical perspective is shown for a cross-domain example of a multi-host system provided in this application.

[0042] Figure 3 A schematic diagram of the FM management topology for a cross-domain example of a multi-host system provided in this application is shown.

[0043] Figure 4 This diagram illustrates the address translation and reason information conversion logic of a cross-domain example of a multi-host system provided in this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] For inter-domain communication in multi-host systems, the industry commonly relies on NTB (Non-transparent Bridge) technology. NTB, as a solution for multi-host systems, is used to facilitate communication between PCI Express domains. A key issue to address in cross-domain communication is address translation between the two domains. Using NTB requires various translation tasks between the two domains, as well as handling packet splitting and reassembly. This involves considerations of hardware implementation complexity, resources, area, and performance. Furthermore, with the increasing computing demands across various fields, not only are the requirements for bandwidth and latency rising, but the topology requirements are also becoming increasingly complex. Traditional NTB methods have significant shortcomings in terms of bandwidth, latency, and topology complexity. For example, address translation and domain configuration issues must be considered in cross-domain paths, increasing latency during data transmission, and address translation requires additional address translation table entries. Using symmetric NTB methods creates areas invisible to the host, consuming dedicated physical ports and wasting system bandwidth. In addition, the NTB method has limitations on topology; it can only rely on the PCIe tree structure topology, and other topologies are not applicable.

[0046] Based on the above analysis, this application proposes a multi-host system and a cross-domain access method. This cross-domain access method is implemented on the multi-host system and is based on the CXL protocol. During the cross-domain process, firstly, the FM (fabrication manager) of the multi-host system supports user-defined commands when managing the topology. This application uses user-defined commands to obtain the DPA (Device physical address, hereinafter referred to as the physical address of the host to be crossed) space of the cross-domain EP device (i.e., PCIe device). Secondly, instead of setting an address translation table, this application proposes a hardware implementation of HPA (Host physical address, hereinafter referred to as the physical address of the host to be crossed) to DPA mapping to complete the address translation on the cross-domain path. Finally, this application also proposes that the FM uniformly manages the allocation of all segment numbers in the topology, associating segment numbers with host domains; that is, each host in each host domain is assigned a unique segment number, and the segment numbers are different for different host domains. Meanwhile, a segment / PID (port identification number) mapping table is implemented on the DSP (Down Stream Port, a type of edge port, hereinafter referred to as the downlink edge port) of the downstream edge switch at the destination end. Using the PBR (Port Based Routing) method, UIO (Userspace I / O) is used to ensure that the packet is correctly forwarded to the vUSP (vUpper Stream Port, a type of edge port, hereinafter referred to as the uplink edge port) at the source end.

[0047] For example, a multi-host system proposed in this application includes: a structure manager and at least two host domains; the host domain includes a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device;

[0048] The structure manager is used to discover and manage the topology of the multi-host system, and to assign segment sequence numbers to hosts in the host domain and port identifiers to the host edge switches, downstream edge switches and edge ports based on the topology; the structure manager is also used to obtain the device information of the terminal device and send the device information to the host that needs to access the terminal device across domains;

[0049] During cross-domain access, the host requiring cross-domain access sends a request message to the terminal device being accessed via the port-based routing and switching network through the host edge switch.

[0050] After receiving the request message, the downstream edge switch associated with the terminal device being accessed across domains establishes an association between the segment sequence number and port identifier carried in the request message, maps the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and forwards it to the terminal device being accessed across domains. The downstream edge switch associated with the terminal device being accessed across domains also receives a completion message returned by the terminal device being accessed across domains, and determines the port identifier corresponding to the segment sequence number carried in the completion message based on the association, and carries the port identifier corresponding to the segment sequence number carried in the completion message as the destination identifier in the completion message, which is then sent to the corresponding host by the port-based routing and switching network.

[0051] For example, the topology of a multi-host system is not limited to a tree structure, but can also be a ring topology, etc. Terminal devices (EP devices) can be PCIe devices. The structure manager assigns unique segment numbers to hosts in different host domains; hosts in different host domains have different segment numbers. The structure manager also assigns unique port identifiers to switches (including host edge switches and downstream edge switches; for ease of description, unless otherwise specified, switches can be understood as including host edge switches and downstream edge switches) and terminal devices. The port identifiers of different switches and terminal devices in different host domains are also different. It should be noted that in the actual physical structure, the port identifier of a switch is actually the port identifier at the uplink edge port vUSP of the switch, while the port identifier of a terminal device is the port identifier of the downlink transformation port DSP on the downstream edge switch connected to the terminal device.

[0052] For example, the device information of the terminal device may include, but is not limited to, the port identifier of the terminal device and the size of the DPA (Dual Address Port) space.

[0053] The following is based on Figure 1 and Figure 2 For example, this section explains the physical topology of a multi-host system and the logical topology for implementing a cross-domain example. Figure 1 The image shows the physical topology for a cross-domain example. Figure 2 This is the topology from a logical perspective for cross-domain examples. For example... Figure 1 and Figure 2As shown, Host0 and Host1 are hosts in two host domains. The terminal devices under the Host0 host domain include EP0, EP1, and EP2. EP0 is directly connected to the host edge switch, therefore this host edge switch can also be called the downstream edge switch of EP0. It should be noted that the switch directly associated with the host is called the host edge switch, and the switch directly associated with the terminal device is called the downstream edge switch. In some cases, the host edge switch and the downstream edge switch can be the same. The terminal device under the Host1 host domain includes EP3. Assume that in this topology, there are three cross-domain paths: 1 represents Host1 accessing EP0 under the Host0 domain; 2 represents Host0 accessing EP3 under Host1; and 3 represents EP1 under the Host0 domain accessing EP3 under the Host1 domain.

[0054] In some optional implementations, the device information includes the DPA space size of the terminal device; the structure manager uses a custom command to read the DPA space size of the terminal device through the component command interface of the downstream edge switch;

[0055] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the terminal device's DPA and latches the base address of the cross-domain host physical address of the terminal device's DPA carried in the configuration message.

[0056] The edge port of the downstream edge switch maps the physical address of the host to be accessed across domains carried in the request message to the physical address of the host to be accessed across domains of the terminal device based on the base address.

[0057] In the CXL protocol, the FM (fabric manager) uses the FM API command set to discover and manage the topology of the PBRfabric (port-based routed switching network). The CXL protocol also supports user-defined commands, increasing management flexibility.

[0058] In this embodiment, the user-defined command supported by the CXL protocol is used to obtain the DPA space size of the EP device that needs to be accessed across domains. The user-defined command is sent to the CCI interface (command component interface) of the downstream edge switch in the form of a UIO request message. The CCI interface converts the UIO message into a CfgRd0 / CfgWr0 command and then reads and writes the configuration space of the EP device to obtain the DPA space size.

[0059] For example, after receiving a custom command, the component command interface of the downstream edge switch converts the custom command into a configuration read / write message and forwards the configuration read / write message to the downlink edge port of the connected terminal device. Upon receiving the configuration read / write message, the terminal device returns a completion message containing the DPA space size to the component command interface. The component command interface converts the completion message into a response to the aforementioned custom command and returns it to the structure manager. The structure manager forwards the DPA space size to the host requiring cross-domain access, and the host requiring cross-domain access allocates a physical address to it based on the DPA space size.

[0060] During the enumeration process, the host being accessed across domains uses configuration messages to read and write the DPA space size of the terminal device, and configures the base address of its DPA in the address space of the host being accessed across domains via configuration write messages. The downlink edge port connected to the terminal device captures this base address of the DPA in the address space of the host being accessed across domains from the configuration write message, uses it as the physical address of the terminal device in the address space of the host being accessed across domains, and latches this base address in the downlink edge port. The address space of the host being accessed across domains can be understood as the host address space of the host domain where the terminal device being accessed across domains resides. The address space of the host requiring cross-domain access is the host address space of the host domain where the host that needs to access the terminal device across domains resides.

[0061] For example, a hardware detection logic is set up at the downlink edge port DSP of the downstream edge switch connected to the EP device. When the DSP forwards the aforementioned CfgRd0 / CfgWr0 commands, if the hardware detection logic detects a write operation to the EP device's Bar register, it latches the value of the write Bar register and uses it as the base address of the EP device's DPA. In subsequent cross-domain access, after the downstream edge switch receives a UIO request message from the host, it can map the HPA address in the UIO request message to the DPA address of the EP device based on this base address. In this way, after the UIO request message reaches the DSP of the downstream edge switch of the EP device being accessed across domains, the hardware mapping logic from HPA to DPA is implemented, thereby completing the address translation on the cross-domain path.

[0062] In some optional implementations, the host requiring cross-domain access is also configured to configure a routing table for the host edge switch requiring cross-domain access. The routing table includes a first mapping relationship between the physical address of the host requiring cross-domain access and the port identifier of the edge port connected to the terminal device that can be accessed across domains, and a second mapping relationship between the port identifier and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0063] For example, in order to enable a host to access EP devices in other host domains across domains through a port-based routing and switching network, the host that needs to access across domains can also configure a FAST (farbricaddress segment table) routing table on its associated host edge switch. This routing table is the routing relationship between HPA and DPID (destination PID), and between DPID and the uplink edge port (egress port) of the host edge switch (a port includes an inbound port and an outbound port).

[0064] In some optional implementations, the first mapping relationship is the mapping relationship between the high-order address of the physical address of the host that needs to cross domains and the port identification number; the low-order address of the physical address of the host that needs to cross domains is the offset address of the terminal device corresponding to the port identification number.

[0065] For example, a mapping table from HPA to DPA is implemented at the downlink edge port DSP location of the downstream edge switch. The HPA address (HPA[51:36], i.e., the high-order address), is mapped to the DPID, as shown in the FAST table. The HPA[35:0], i.e., the low-order address, is the offset address of the EP device's memory space.

[0066] In some alternative implementations, the downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host to be accessed across domains carried in the request message with the base address of the physical address of the host to be accessed across domains of the terminal device being accessed across domains, thereby realizing the address mapping from the physical address of the host to the physical address of the host to be accessed across domains.

[0067] For example, when mapping HPA to DPA, the high-order address, i.e., HPA[51:36], can be replaced with the base address of the EP device being accessed across domains, thus forming the DPA address of the EP device being accessed across domains together with the offset address in the low-order address, i.e., HPA[35:0]. As mentioned above, the base address of the EP device is latched by the hardware detection logic set at the DSP of the downstream edge switch when the structure manager accesses the DPA space size of the EP device. In this way, the mapping from HPA to DPA is implemented at the DSP of the downstream edge switch during the cross-domain access process.

[0068] In some optional implementations, the request message carries the physical address of the host to be accessed across domains and the port identifier of the terminal device to be accessed across domains; after receiving the request message from the host, the host edge switch determines the edge port through the routing table and transmits the request message from the edge port to the port-based routing and switching network.

[0069] For example, the request message carries the host's HPA and the port identifier (DIP) of the EP device being accessed across domains (i.e., the Destination Identifier, which is the port identifier assigned to the downlink edge port connected to the EP device). Upon receiving the request message, the host edge switch can parse the HPA and DIP, further mapping them to the outbound port. It then assembles the request message into a PBR packet and transmits it to the port-based routing and switching network. This network forwards the PBR packet to the downstream edge switch associated with the EP device being accessed across domains. Upon receiving the PBR packet, the downstream edge switch parses it, removes the DIP, converts the HPA address to the DPA, assembles it into an HBR packet, and transmits it to the corresponding EP device.

[0070] In some alternative implementations, when the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host being accessed across domains.

[0071] For example, during the initialization of the downstream edge switch, all EP devices that need to cross domains are set to the unbound state, meaning that the Host of the host domain where the EP device is located is initially invisible.

[0072] This application embodiment uses user-defined commands to obtain the DPA space size of the cross-domain EP device. After the UIO request message used by the host for cross-domain access reaches the DSP of the downstream edge switch associated with the EP device, the hardware implementation of HPA to DPA mapping is used to complete the address translation on the cross-domain path. At the same time, this application embodiment also utilizes the allocation of segment sequence numbers of all host domains in the FM unified management topology to associate segment sequence numbers with host domains, and the segment sequence numbers of different hosts do not overlap. The segment / t / PID mapping table is implemented on the DSP of the destination switch to obtain the destination identifier. The PBR routing method is used to complete the correct forwarding of the UIO completion message to the source vUSP.

[0073] The following uses Figure 1 and Figure 2 The topology shown illustrates the relevant implementation details of cross-domain path management in the embodiments of this application.

[0074] 1. During downstream edge switch initialization, all EP devices requiring cross-domain access are set to an unbound state, meaning they are initially invisible to hosts within the host domain of the EP device. For example... Figure 1 and Figure 2 The EP devices EP0, EP1, EP2 and EP3 shown are all devices that need to cross domains. They can be set to be invisible to host0 for EP0, and invisible to host1 for EP3.

[0075] 2. FM can use the crawl out command to discover the entire topology and use the PID assignment command to complete the PID assignment for all devices in the topology. For example... Figure 1 , Figure 2 and Figure 3 As shown, the uplink edge ports vUSP of switch0 and switch1 have PIDs of 0 and 1, respectively. The downlink edge ports DSP connecting EP0, EP1, and EP3 have PIDs of 8, 5, and 7, respectively.

[0076] 3. FM uses user-defined commands to read and write the configuration space of the PCIe device via the CCI interface of the switch using the CfgRd0 / CfgWr0 commands:

[0077] Read and write the Bar register of the PCIe device to obtain the DPA space size of the EP device.

[0078] User-defined Opcodes are C000h-FFFFh.

[0079] The specific details of the user-defined commands in this application are shown in the table below:

[0080]

[0081]

[0082] In the table, setting the register number to 10h, 14h, 18h or 1ch indicates that reading or writing to this address will yield the size of the memory space requested by each Bar register space of the EP device, i.e., the DPA size.

[0083] 4. After collecting information from all EP devices, FM exchanges information with the hosts in the host domain. For example... Figure 1 , Figure 2 and Figure 3The host informs Host0 of the DPA space size of Host1.EP1 and Host1 of the DPA space size of Host0.EP0. Based on the cross-domain device DPA space size, the host allocates a cross-domain address space (GIM) within the address range of fabric_base[51:36] and fabric_limit[51:36], and this address space can be accessed using UIO packets. Simultaneously, the host configures the FAST routing table in the host edge switch and / or downstream edge switches. This table represents the routing relationships between HPA and DPID, and between DPID and egress port (outbound port), following the CXL protocol. Figure 3 As shown, a FAST routing table is configured on the host edge switch Switch1 for cross-domain path 1, on the host edge switch Switch0 for cross-domain path 2, and on the downstream edge switch Switch2 for cross-domain path 3. The FAST routing table is configured by the host in the domain where the switch is located.

[0084] 5. FM assigns segment sequence numbers to hosts in the host domain. The segment sequence numbers between host domains do not overlap. The purpose of this is to ensure that the UIO completion message returned by the EP device can be correctly routed to each host.

[0085] The details of using HPA / DPA translation logic and segment / PID mapping tables to complete address translation and routing information conversion on cross-domain paths are as follows: See below. Figure 4 As shown:

[0086] 1. The CCI interfaces in the host edge switch and downstream edge switches can receive and parse user-defined commands from the FM. The user-defined commands specify the outbound port number of the current-level switch (port ID in the table) and the offset address of the configuration space register of each EP device. The hardware CCI interface can translate the user-defined commands into CfgWr0 / CfgRd0 commands, thereby accessing the configuration space of the EP device. This process can be found in the description of user-defined commands above and will not be repeated here.

[0087] 2. Implement the HPA2DPA mapping table at the DSP location of the downstream edge switch. The HPA addresses HPA[51:36] are mapped to DPID, as shown in the FAST table. HPA[35:0] represents the offset address of the EP device's memory space.

[0088] At the DSP of the downstream edge switch, implement the hardware detection logic for writing to the Bar register of the EP device. When the DSP forwards CfgWr0 (the Bar register of Ep3) (indicating a write operation to the Bar register of Ep3), if it detects that it is writing to the Bar register, it latches the value to be written and uses it as the base address of the DPA of the EP device.

[0089] 3. After the UIO request message with DPID is forwarded to the DSP of the downstream edge switch, the DPID information is stripped, and the HPA is mapped to DPA (that is, the base address of DPA + offset), completing the conversion from PBR packet to HBR (hierarchy based routing) packet.

[0090] 4. The downstream edge switch's DSP will also establish a segment / PID table. The DSP will obtain the SPID in the UIO request message and store the association between the SPID and the segment in the UIO request message in the segment / PID table.

[0091] After the downstream edge switch transmits the HBR packet to the EP device via the DSP, the EP device will generate a UIO completion message for the HBR packet. The UIO completion message will carry the destination segment (the segment number to be returned to the host).

[0092] After the downstream edge switch receives the UIO completion message from the EP device through the DSP, it first looks up the segment / PID table. If the segment / PID table is matched, it obtains the corresponding PID from the table based on the destination segment. This PID can be used as the SPID to form a PBR packet. The PBR packet with the group number is then forwarded to the source host (i.e., the host corresponding to the SPID) for forwarding, thus completing the cross-domain access.

[0093] This application uses FM to uniformly manage the entire topology and obtains the DPA space size of cross-domain devices based on user-defined commands, preparing for the establishment of GIM address spaces for hosts. PBR routing is used on cross-domain paths, freeing cross-domain communication from being limited to the tree-like topology of PCIe, enabling cross-domain communication in more complex topologies.

[0094] On the cross-domain device side, the DSP above the device implements the segment / PID mapping table, cross-domain address mapping conversion logic, and UIO request message processing logic. This processing logic is used to extract the DPID information from the UIO request message (the purpose is to convert the PBR packet into an HBR packet) and replace the HPA. This processing logic is simple to implement and does not introduce significant overhead in terms of resources and area.

[0095] Based on the CXL protocol and following the PBR routing method, this application allows cross-domain paths and non-cross-domain paths to share the physical port (physical path), eliminating the need to allocate a dedicated physical port for cross-domain paths.

[0096] Accordingly, this application provides a cross-domain access method in a second aspect.

[0097] The cross-domain access method is implemented in a multi-host system, which includes: a structure manager and at least two host domains; each host domain includes a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the structure manager is used to discover and manage the topology of the multi-host system, and assign segment sequence numbers to hosts in the host domains and assign port identifiers to the host edge switches, downstream edge switches, and terminal devices based on the topology; the structure manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains; the cross-domain access method includes:

[0098] During cross-domain access, the host requiring cross-domain access sends a request message to the terminal device being accessed via the associated host edge switch through a port-based routing and switching network.

[0099] After receiving the request message, the downstream edge switch associated with the terminal device being accessed across domains establishes the association between the segment sequence number and port identifier carried in the request message, maps the physical address of the host to be accessed across domains in the request message to the physical address of the host being accessed across domains, and then forwards it to the terminal device being accessed across domains.

[0100] When the downstream edge switch associated with the terminal device being accessed across domains receives the completion message returned by the terminal device being accessed across domains, and determines the port identifier corresponding to the segment sequence number carried in the completion message based on the association relationship, the port identifier corresponding to the segment sequence number carried in the completion message is carried as the destination identifier in the completion message and sent to the corresponding host by the port-based routing and switching network.

[0101] In some alternative implementations,

[0102] The device information includes the DPA space size of the terminal device; the method further includes:

[0103] The structure manager sends a custom command to the component command interface of the downstream edge switch to obtain the DPA space size of the terminal device;

[0104] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the terminal device's DPA and latches the base address of the physical address of the cross-domain host carrying the DPA of the terminal device in the configuration message.

[0105] The step of mapping the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwarding it to the terminal device being accessed across domains, includes:

[0106] The edge port of the downstream edge switch maps the physical address of the host requiring cross-domain access carried in the request message to the physical address of the host requiring cross-domain access of the terminal device being accessed across domains, based on the base address.

[0107] In some alternative implementations, the method further includes:

[0108] The host requiring cross-domain access is also configured to configure a routing table for the host edge switch requiring cross-domain access. The routing table includes a first mapping relationship between the physical address of the host requiring cross-domain access and the port identifier of the terminal device that can be accessed across domains, and a second mapping relationship between the port identifier and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0109] In some optional implementations, the first mapping relationship is the mapping relationship between the high-order address of the physical address of the host that needs to cross domains and the port identification number; the low-order address of the physical address of the host that needs to cross domains is the offset address of the terminal device corresponding to the port identification number.

[0110] In some optional implementations, the step of mapping the physical address of the host to be accessed across domains in the request message to the physical address of the host being accessed across domains, and then forwarding it to the terminal device being accessed across domains, includes:

[0111] The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host to be accessed across domains carried in the request message with the base address of the physical address of the host to be accessed across domains of the terminal device being accessed across domains, thereby realizing the address mapping from the physical address of the host to the physical address of the host to be accessed across domains.

[0112] In some optional implementations, the request message carries the physical address of the host requiring cross-domain access and the port identifier of the edge port connecting the terminal device being accessed across domains; the method further includes:

[0113] After receiving the request message from the host, the host edge switch determines the edge port through the routing table and transmits the request message from the edge port to the port-based routing and switching network.

[0114] In some alternative implementations, the method further includes:

[0115] When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host being accessed across domains.

[0116] The above method is implemented in the multi-host system provided in the above embodiments. For specific implementation details, please refer to the description in the embodiments of the multi-host system above, which will not be repeated here.

[0117] It is understood that the circuit structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0118] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-host system, characterized by, The application relates to a multi-host system, which comprises a structure manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device and a downstream edge switch associated with the terminal device; the structure manager is used for discovering and managing the topology of the multi-host system, and assigning a segment serial number to the host in the host domain and a port identification number to the host edge switch, the downstream edge switch and the terminal device based on the topology; the structure manager is also used for obtaining device information of the terminal device and sending the device information to the host needing to access the terminal device across domains; wherein the host in each host domain is assigned a unique segment serial number, and the segment serial numbers of different host domains are different; in the cross-domain access process, the host needing to access the terminal device across domains sends a request message to the terminal device accessed across domains through the host edge switch and a port-based routing switching network; the downstream edge switch associated with the terminal device accessed across domains receives the request message, establishes an association between the segment serial number and the port identification number carried in the request message, maps the physical address of the host needing to access the terminal device across domains into the physical address of the host accessed across domains, and then forwards the request message to the terminal device accessed across domains; the downstream edge switch associated with the terminal device accessed across domains also receives a completion message returned by the terminal device accessed across domains, determines the port identification number corresponding to the segment serial number carried in the completion message based on the association, and carries the port identification number corresponding to the segment serial number in the completion message as a destination identification in the completion message, and then sends the completion message to the corresponding host through the port-based routing switching network; wherein a segment serial number and port identification number mapping table is realized at the downstream port of the downstream edge switch, so that the completion message is correctly forwarded to the source end by using the port-based routing mode. The device information comprises the DPA space size of the terminal device; the structure manager obtains the DPA space size of the terminal device through a component command interface of the downstream edge switch by using a self-defined command; An edge port connected to the terminal device on the downstream edge switch captures a configuration message of the DPA of the terminal device, and latches the base address of the physical address of the terminal device's DPA carried in the configuration message; The edge port of the downstream edge switch maps the physical address of the host needing to access the terminal device across domains carried in the request message into the physical address of the host accessed across domains of the terminal device based on the base address. The host needing to access the terminal device across domains is also used for configuring a routing table for the host edge switch needing to access the terminal device across domains; the routing table comprises a first mapping relationship between the physical address of the host needing to access the terminal device across domains and the port identification number of the edge port connected to the terminal device capable of being accessed across domains, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host needing to access the terminal device across domains.

2. The multi-host system of claim 1, wherein, ​ ​ ​ 3. The multi-host system according to claim 1 or 2, characterized in that, ​ 4. The multi-host system of claim 3, wherein, The first mapping relationship is a mapping relationship between a high-order address of the cross-domain required host physical address and the port identification number; and a low-order address of the cross-domain required host physical address is an offset address of the terminal device corresponding to the port identification number.

5. The multi-host system of claim 4, wherein, The downstream edge switch associated with the terminal device accessed across domains replaces the high-order address of the cross-domain required host physical address carried by the request message with a base address of the cross-domain host physical address of the terminal device accessed across domains, so as to realize address mapping of the cross-domain required host physical address to the cross-domain host physical address.

6. The multi-host system of claim 3, wherein, The request message carries a cross-domain required host physical address of the host accessed across domains and the port identification number of the terminal device accessed across domains; after receiving the request message from the host, the host edge switch determines the outbound port through the routing table, and transmits the request message to the port-based routing switching network from the outbound port.

7. The multi-host system according to any of claims 1, 2, 4-6, characterized by When the downstream edge switch is initialized, the terminal device accessed across domains is configured to be invisible to the host accessed across domains.

8. A cross-domain access method, characterized by, The cross-domain access method is implemented in a multi-host system, and the multi-host system includes a structure manager and at least two host domains; the host domain includes a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the structure manager is used to discover and manage a topology of the multi-host system, and assign a segment serial number to the host in the host domain and assign a port identification number to the host edge switch, the downstream edge switch, and the terminal device based on the topology; the structure manager is also used to obtain device information of the terminal device, and send the device information to the host requiring to access the terminal device across domains; wherein the host in each host domain is assigned a unique segment serial number, and the segment serial numbers of different host domains are different; the cross-domain access method includes: During the cross-domain access process, the host requiring to access across domains sends a request message to the terminal device accessed across domains via the port-based routing switching network through the associated host edge switch; After receiving the request message, the downstream edge switch associated with the terminal device accessed across domains establishes an association relationship between the segment serial number and the port identification number carried in the request message, maps the cross-domain required host physical address in the request message to a cross-domain host physical address, and then forwards to the terminal device accessed across domains. The downstream edge switch associated with the terminal device accessed cross-domain receives a completion message returned by the terminal device accessed cross-domain, and determines a port identification number corresponding to a segment serial number carried in the completion message based on the association relationship, and carries the port identification number corresponding to the segment serial number in the completion message as a destination identification in the completion message, and sends the completion message to a corresponding host by a port-based routing switching network; wherein a segment serial number and port identification number mapping table is implemented on a downstream port of the downstream edge switch, so that the completion message is correctly forwarded to a source end by using a port-based routing mode.

9. The cross-domain access method of claim 8, wherein, The device information includes a DPA space size of the terminal device; the method further includes: The structure manager sends a custom command to a component command interface of the downstream edge switch to obtain the DPA space size of the terminal device; An edge port of the downstream edge switch connected to the terminal device captures a configuration message of a DPA of the terminal device, and latches a base address of a cross-domain host physical address of the DPA carried in the configuration message; The method further includes: The edge port of the downstream edge switch maps a cross-domain host physical address carried in the request message to a cross-domain host physical address of the terminal device accessed cross-domain based on the base address.

10. The cross-domain access method of claim 8 or 9, characterized by, The method further includes: The host needing cross-domain access is further configured to configure a routing table for a host edge switch needing cross-domain access, the routing table including a first mapping relationship between a cross-domain host physical address of the host needing cross-domain access and a port identification number of the terminal device capable of being accessed cross-domain, and a second mapping relationship between the port identification number and an outbound port of the host edge switch associated with the host needing cross-domain access.

11. The cross-domain access method of claim 10, wherein, The first mapping relationship is a mapping relationship between a high address of the cross-domain host physical address and the port identification number; a low address of the cross-domain host physical address is an offset address of the terminal device corresponding to the port identification number.

12. The cross-domain access method of claim 11, wherein, The method further includes: The downstream edge switch associated with the terminal device accessed cross-domain replaces a high address of the cross-domain host physical address carried in the request message with a base address of a cross-domain host physical address of the terminal device accessed cross-domain, so as to realize address mapping of the cross-domain host physical address to the cross-domain physical address.

13. The cross-domain access method of claim 11, wherein, The request message carries a cross-domain host physical address of the host accessed cross-domain and the port identification number of the edge port connected to the terminal device accessed cross-domain; the method further includes: After the host edge switch receives the request packet from the host, the host edge switch determines the egress port through the routing table and transmits the request packet from the edge port to the port-based routing switch network.

14. The cross-domain access method according to any one of claims 8, 9, 11-13, characterized by, The method further comprises: The terminal device to be accessed across the domain is configured to be invisible to the host being accessed across the domain when the downstream edge switch is initialized.

Images