Multi-host system and cross-domain access method

Abstract

The invention provides a multi-host system and a cross-domain access method. The system comprises a structure manager and at least two host domains, the structure manager is used for discovering and managing a topological structure of the multi-host system, distributing segment serial numbers for hosts and distributing port identification numbers for host edge switches, edge ports and downstream edge switches; the structure manager obtains equipment information of the terminal equipment and sends the equipment information to the host; the host sends a request message to the terminal equipment through the host edge switch via the port-based routing switching network; the downstream edge switch establishes an association relationship between a segment serial number and a port identification number carried in the request message, and maps a physical address of a host needing cross-domain to a physical address of a host subjected to cross-domain; and the downstream edge switch receives the completion message, determines a port identification number corresponding to the segment serial number carried in the completion message based on the association relationship, carries the port identification number in the completion message as a destination end identifier, and sends the port identification number to a corresponding host.

Description

Technical Field

[0001] The present application belongs to the field of chip technology, and in particular relates to a multi-host system and a cross-domain access method. Background Art

[0002] PCIe is a high-speed serial computer expansion bus standard that mainly provides a bus interface for the CPU to access peripheral devices. The PCIe topology is a tree topology, and each host manages its own tree topology. Each host controls its own downstream port to form an independent address domain. The addresses between hosts are completely isolated and cannot access each other. With the continuous development of technology, a single host can no longer meet the application of the system, and the application requirements of multiple hosts are increasing. Multi-host allows multiple computing nodes (hosts) to share and access one or more common hardware resources. In this way, resource utilization is improved, the flexibility and scalability of the system are enhanced, and load management and fault recovery strategies are supported. Realizing cross-domain access between multiple hosts and multiple devices has become a topic that must be studied. Summary of the invention

[0003] The purpose of this application is to provide a multi-host system and a cross-domain access method, aiming to solve the technical problems of low performance of cross-domain access terminal devices and dependence on tree topology structure in the multi-host system.

[0004] According to a first aspect of the present application, a multi-host system is provided, comprising: a fabric manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device;

[0005] The structure manager is used to discover and manage the topology of the multi-host system, and allocate segment numbers to hosts in the host domain and port identification numbers to the host edge switch, downstream edge switch and terminal device based on the topology; the structure manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains;

[0006] During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device to be accessed across the domain through the host edge switch and the port-based routing switching network;

[0007] After receiving the request message, the downstream edge switch associated with the terminal device that is accessed across domains establishes an association relationship between the segment number and the port identification number carried in the request message, and maps the physical address of the host that needs to cross domains in the request message to the physical address of the host that is accessed across domains, and then forwards it to the terminal device that is accessed across domains; the downstream edge switch associated with the terminal device that is accessed across domains also receives a completion message returned by the terminal device that is accessed across domains, and after determining the port identification number corresponding to the segment number carried in the completion message based on the association relationship, carries the port identification number corresponding to the segment number carried in the completion message as the destination identifier in the completion message, and sends it to the corresponding host by the port-based routing switching network.

[0008] In an optional implementation, the device information includes the DPA space size of the terminal device: the fabric manager obtains the DPA space size of the terminal device through a component command interface of the downstream edge switch using a custom command;

[0009] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message;

[0010] The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device accessed across domains based on the base address.

[0011] In an optional implementation, the host requiring cross-domain access is also used to configure a routing table for the host edge switch requiring cross-domain access, the routing table including a first mapping relationship between the cross-domain host physical address of the host requiring cross-domain access and the port identification number of the edge port connected to the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0012] In an optional implementation, the first mapping relationship is a mapping relationship between a high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is an offset address of the terminal device corresponding to the port identification number.

[0013] In an optional implementation, the downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host required to cross domains carried in the request message with the base address of the physical address of the host being accessed across domains of the terminal device being accessed across domains, thereby realizing address mapping from the physical address of the host required to cross domains to the physical address of the host being accessed across domains.

[0014] In an optional implementation, the request message carries the physical address of the host requiring cross-domain access and the port identification number of the terminal device being accessed across domains; after the host edge switch receives the request message from the host, it determines the edge port through the routing table, and transmits the request message from the edge port to the port-based routing switching network.

[0015] In an optional implementation, when the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

[0016] According to a second aspect of the present application, a cross-domain access method is provided, which is implemented in a multi-host system, wherein the multi-host system comprises: a fabric manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the fabric manager is used to discover and manage the topology of the multi-host system, and based on the topology, allocates segment numbers to the hosts in the host domain and allocates port identification numbers to the host edge switch, the downstream edge switch, and the terminal device; the fabric manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains; the cross-domain access method comprises:

[0017] During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device that is accessed across the domain through the associated host edge switch via the port-based routing switching network;

[0018] After receiving the request message, the downstream edge switch associated with the terminal device accessed across domains establishes an association relationship between the segment sequence number and the port identification number carried in the request message, and maps the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwards it to the terminal device accessed across domains;

[0019] The downstream edge switch associated with the terminal device accessed across domains receives the completion message returned by the terminal device accessed across domains, and determines the port identification number corresponding to the segment number carried in the completion message based on the association relationship. Then, the port identification number corresponding to the segment number carried in the completion message is carried as the destination identifier in the completion message, and is sent to the corresponding host by the port-based routing switching network.

[0020] In an optional implementation manner, the device information includes a DPA space size of the terminal device; and the method further includes:

[0021] The fabric manager sends a custom command to the component command interface of the downstream edge switch to obtain the DPA space size of the terminal device;

[0022] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message;

[0023] The step of mapping the cross-domain host physical address in the request message to the cross-domain host physical address and then forwarding the physical address to the terminal device accessed across the domain includes:

[0024] The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device accessed across domains based on the base address.

[0025] In an optional implementation, the method further includes:

[0026] The host that requires cross-domain access is also used to configure a routing table for the host edge switch that requires cross-domain access, and the routing table includes a first mapping relationship between the cross-domain host physical address of the host that requires cross-domain access and the port identification number of the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host that requires cross-domain access.

[0027] In an optional implementation, the first mapping relationship is a mapping relationship between a high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is an offset address of the terminal device corresponding to the port identification number.

[0028] In an optional implementation, mapping the cross-domain host physical address in the request message to the cross-domain host physical address and then forwarding it to the terminal device accessed across domains includes:

[0029] The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the host physical address required to be accessed across domains carried in the request message with the base address of the host physical address of the terminal device being accessed across domains, thereby realizing address mapping from the host physical address required to be accessed across domains to the physical address to be accessed across domains.

[0030] In an optional implementation, the request message carries the cross-domain host physical address of the host to be accessed across domains and the port identification number of the edge port connected to the terminal device to be accessed across domains; the method further includes:

[0031] After receiving the request message from the host, the host edge switch determines the edge port through the routing table, and transmits the request message from the edge port to the port-based routing switching network.

[0032] In an optional implementation, the method further includes:

[0033] When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

[0034] Compared with the related art, the technical solution of this application has the following advantages:

[0035] This application manages the entire topology in a unified manner based on FM, obtains the DPA space size of the cross-domain device according to the user-defined command, and prepares for the host to establish the GIM (global integrated memory) address space. PBR routing is used on the cross-domain path, and cross-domain is no longer limited to the tree topology of PCIe, and cross-domain can be realized in more complex topologies.

[0036] On the cross-domain device side, the DSP above the device implements the segment / PID mapping table, the cross-domain address mapping relationship conversion logic, and the UIO request message processing logic, which is used to strip the DPID information in the UIO request message (the purpose is to convert the PBR packet into the HBR packet) and replace the HPA. The processing logic is simple to implement and does not introduce too much overhead in terms of resources and area.

[0037] This application is based on the CXL protocol and the PBR routing method. On the cross-domain path, the cross-domain path and the non-cross-domain path can share the physical port, that is, the physical path can be shared without the need to allocate a dedicated physical port for the cross-domain path.

[0038] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures and processes indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of the physical topology structure of a cross-domain example of a multi-host system provided by the present application is shown.

[0041] Figure 2 A schematic diagram of a topological structure from a logical perspective of a multi-host system cross-domain example provided by the present application is shown.

[0042] Figure 3 A schematic diagram of the FM management topology structure of a multi-host system cross-domain example provided by the present application is shown.

[0043] Figure 4 A logical diagram of address translation and reason information conversion according to a multi-host system cross-domain example provided by the present application is shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The industry generally needs to use NTB (Non-transparent Bridge) technology for inter-domain communication of Multi-Host (multi-host system). NTB is a solution for Multi-Host system and is used to complete communication between PCIExpress domains. Among them, an important problem to be solved in cross-domain is that the addresses between the two domains need to be translated; using the NTB method, various translations between the two domains need to be performed, as well as the problem of unpacking and assembling packets. It will involve considerations of hardware implementation complexity, resources, area, performance, etc. In addition, with the growing computing needs in various fields, not only the requirements for bandwidth and latency are getting higher and higher, but also the requirements for topology are becoming more and more complex. The traditional NTB method has obvious shortcomings in bandwidth, latency and topological complexity. For example, address translation and configuration issues between domains need to be considered on the cross-domain path. During data transmission, latency will be increased, and address translation also requires additional address translation entries. Using symmetrical NTB, there will be areas that are invisible to the host, which will occupy dedicated physical ports. From the system point of view, it will occupy the bandwidth of a physical port, wasting system bandwidth. In addition, the NTB method has restrictions on the topology structure and can only rely on the PCIE tree structure topology. Other topologies are not applicable.

[0046] Based on the above analysis, the present application proposes a multi-host system and a cross-domain access method. The cross-domain access method is implemented on the multi-host system and is based on the CXL protocol. In the cross-domain process, first, when the FM (fabricmanager) of the multi-host system manages the topology, it supports user-defined commands. The present application uses user-defined commands to obtain the DPA (Device physical address, device physical address, hereinafter referred to as the cross-domain host physical address) space size of the cross-domain EP device (that is, PCIe device). Secondly, the address translation table is no longer set. The present application proposes a hardware implementation method of HPA (Host physical address, host physical address, hereinafter referred to as the cross-domain host physical address) to DPA mapping to complete the address translation on the cross-domain path. Finally, the present application also proposes that FM uniformly manages the allocation of all segment numbers in the topology, and associates the segment number with the host domain, that is, each host in the host domain is assigned a unique segment number, and the segment numbers of different host domains are different. At the same time, a segment / PID (port identification number) mapping table is implemented on the DSP (Down Stream Port, a type of edge port, hereinafter referred to as the downstream edge port) of the downstream edge switch at the destination end, and the PBR (Port Based Routing, port-based routing) method is used to implement UIO (Userspace I / O) to complete the correct forwarding of the message to the source end vUSP (vUpper Stream Port, virtual upstream port, a type of edge port, hereinafter referred to as the upstream edge port).

[0047] Exemplarily, a multi-host system proposed in the present application includes: a fabric manager and at least two host domains; the host domain includes a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device;

[0048] The structure manager is used to discover and manage the topology of the multi-host system, and allocate segment numbers to hosts in the host domain and port identification numbers to the host edge switches, downstream edge switches and edge ports based on the topology; the structure manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains;

[0049] During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device to be accessed across the domain through the host edge switch and the port-based routing switching network;

[0050] After receiving the request message, the downstream edge switch associated with the terminal device that is accessed across domains establishes an association relationship between the segment number and the port identification number carried in the request message, and maps the physical address of the host that needs to cross domains in the request message to the physical address of the host that is accessed across domains, and then forwards it to the terminal device that is accessed across domains; the downstream edge switch associated with the terminal device that is accessed across domains also receives a completion message returned by the terminal device that is accessed across domains, and after determining the port identification number corresponding to the segment number carried in the completion message based on the association relationship, carries the port identification number corresponding to the segment number carried in the completion message as the destination identifier in the completion message, and sends it to the corresponding host by the port-based routing switching network.

[0051] Exemplarily, the topology of a multi-host system is not limited to a tree structure, but can also be a ring topology, etc. The terminal device (EP device) can be a PCIe device. The structure manager assigns unique segment numbers to hosts in different host domains. Hosts in different host domains have different segment numbers. The port identification numbers assigned by the structure manager to switches (including host edge switches and downstream edge switches. For the convenience of description, unless otherwise specified, switches can be understood as including host edge switches and downstream edge switches) and terminal devices are also unique. The port identification numbers of different switches and terminal devices between different host domains are also different. It should be noted that in the actual physical structure, the port identification number of the switch is actually the port identification number at the upstream edge port vUSP of the switch, and the port identification number of the terminal device is the port identification number of the downstream conversion port DSP on the downstream edge switch connected to the terminal device.

[0052] Exemplarily, the device information of the terminal device may include, but is not limited to, a port identification number of the terminal device and a DPA (domain-spanned host physical address) space size.

[0053] Below Figure 1 and Figure 2 As an example, the physical topology of a multi-host system and the topology from a logical perspective of implementing a cross-domain example are described. Figure 1 The figure shows the physical topology of a cross-domain example. Figure 2 This is the topology from the logical perspective of the cross-domain example. Figure 1 and Figure 2As shown, Host0 and Host1 are hosts of two host domains respectively, and the terminal devices under Host0 host domain include EP0, EP1 and EP2, among which EP0 is directly connected to the host edge switch, so the host edge switch can also be called the downstream edge switch of EP0. It should be noted that the switch directly associated with the host is called the host edge switch, and the switch directly associated with the terminal device is called the downstream edge switch. In some cases, the host edge switch and the downstream edge switch can be the same. The terminal devices under Host1 host domain include EP3. Assume that under this topology, there are three cross-domain paths: 1 means Host1 accesses EP0 under Host0 domain; 2 means Host0 accesses EP3 under Host1; 3 means EP1 under Host0 domain accesses EP3 under Host1 domain.

[0054] In some optional implementations, the device information includes the DPA space size of the terminal device; the fabric manager uses a custom command to read the DPA space size of the terminal device through a component command interface of the downstream edge switch;

[0055] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message;

[0056] The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device accessed across domains based on the base address.

[0057] In the CXL protocol, FM (fabric manager) completes the topology discovery and management of PBRfabric (port-based routing switching network) through the FM API command set. The CXL protocol also supports user-defined commands, which increases the flexibility of management.

[0058] In the embodiment of the present application, the user-defined command supported by the CXL protocol is used to obtain the DPA space size of the EP device that needs to be accessed across domains. The user-defined command is sent to the CCI interface (commad component interface, component command interface) of the downstream edge switch in the form of a UIO request message. The CCI interface converts the UIO message into a CfgRd0 / CfgWr0 command, and then reads and writes the configuration space of the EP device to obtain the DPA space size.

[0059] Exemplarily, after receiving the custom command, the component command interface of the downstream edge switch converts the custom command into a configuration read-write message, and forwards the configuration read-write message to the downstream edge port connected to the terminal device. After receiving the configuration read-write message, the terminal device returns a completion message with the DPA space size to the component command interface. The component command interface converts the completion message into a response to the above custom command and returns it to the structure manager. The structure manager forwards the DPA space size to the host that needs to cross the domain, and the host that needs to cross the domain allocates the physical address of the host that needs to cross the domain according to the DPA space size.

[0060] During the enumeration process, the host that is accessed across domains will use the configuration message to read and write the DPA space size of the terminal device, and configure the base address of its DPA in the address space of the host that is accessed across domains for the terminal device through the configuration write message. The downstream edge port connected to the terminal device will capture the base address of the DPA in the address space of the host that is accessed across domains in the configuration write message as the physical address of the terminal device in the address space of the host that is accessed across domains, and lock the base address in the downstream edge port. The host address space that is accessed across domains can be understood as the host address space of the host domain where the terminal device that is accessed across domains is located. The host address space that needs to be accessed across domains is the host address space of the host domain where the host that needs to access the terminal device across domains is located.

[0061] Exemplarily, a hardware detection logic is set at the DSP of the downstream edge port connected to the EP device on the downstream edge switch. When the DSP forwards the above CfgRd0 / CfgWr0 command, if the hardware detection logic detects the action of writing the Bar register of the EP device, the value of writing the Bar register is latched and used as the base address of the DPA of the EP device. In subsequent cross-domain access, after the downstream edge switch receives the UIO request message sent from the host, it can map the HPA address in the UIO request message to the DPA address of the EP device based on the base address. In the above manner, after the UIO request message crosses the domain to the DSP of the downstream edge switch of the EP device being accessed across the domain, the hardware mapping logic from HPA to DPA is implemented, thereby completing the address translation on the cross-domain path.

[0062] In some optional implementations, the host requiring cross-domain access is also used to configure a routing table for the host edge switch requiring cross-domain access, the routing table including a first mapping relationship between the cross-domain host physical address of the host requiring cross-domain access and the port identification number of the edge port connected to the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host requiring cross-domain access.

[0063] Exemplarily, in order to enable the host to access EP devices in other host domains across domains through a port-based routing switching network, the host that requires cross-domain access can also configure a FAST (farbricaddress segmenttable) routing table on its associated host edge switch. The routing table is the routing relationship between the HPA and the DPID (destination PID), and the DPID and the uplink edge port egress port (egress port) of the host edge switch (a port includes an inbound port and an egress port).

[0064] In some optional implementations, the first mapping relationship is a mapping relationship between a high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is an offset address of the terminal device corresponding to the port identification number.

[0065] Exemplarily, at the downstream edge port DSP position of the downstream edge switch, a mapping table from HPA to DPA is implemented. Among them, HPA[51:36] of the HPA address, i.e., the high-order address, has a mapping relationship with DPID, which is reflected in the FAST table. HPA[35:0], i.e., the low-order address, is the offset address (offset) of the memory space of the EP device.

[0066] In some optional implementations, the downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host required to cross domains carried in the request message with the base address of the physical address of the host being accessed across domains of the terminal device being accessed across domains, thereby realizing address mapping from the physical address of the host required to cross domains to the physical address of the host being accessed across domains.

[0067] Exemplarily, when HPA is mapped to DPA, the high-order address, i.e., HPA[51:36], can be replaced with the base address of the EP device being accessed across domains, thereby forming the DPA address of the EP device being accessed across domains together with the low-order address, i.e., the offset address in HPA[35:0]. As described above, the base address of the EP device is latched by the hardware detection logic set at the DSP of the downstream edge switch when the fabric manager accesses the DPA space size of the EP device. In the above manner, during the cross-domain access process, the mapping from HPA to DPA is implemented at the DSP of the downstream edge switch.

[0068] In some optional implementations, the request message carries the physical address of the host requiring cross-domain access and the port identification number of the terminal device being accessed across domains; after the host edge switch receives the request message from the host, it determines the edge port through the routing table, and transmits the request message from the edge port to the port-based routing switching network.

[0069] Exemplarily, the request message will carry the host's HPA and the port identification number of the EP device being accessed across domains (that is, the destination identification DPID, which is the port identification number assigned to the downstream edge port connected to the EP device). After receiving the request message, the host edge switch can parse the HPA and DPID therein, and further map them to the outbound port, so as to form the request message into a PBR packet, and then transmit it to the port-based routing switching network, and the port-based routing switching network will forward the PBR packet to the downstream edge switch associated with the EP device being accessed across domains. After receiving the PBR packet, the downstream edge switch parses the PBR packet, strips off the DPID therein, converts the HPA address into DPA, and then forms an HBR packet and transmits it to the corresponding EP device.

[0070] In some optional implementations, when the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

[0071] Exemplarily, when the downstream edge switch is initialized, the states of all EP devices that need to cross domains are set to unbound states, that is, the host in the host domain where the EP device is located is first invisible.

[0072] The embodiment of the present application uses a user-defined command to obtain the DPA space size of the cross-domain EP device, and after the UIO request message used by the host for cross-domain access reaches the DSP of the downstream edge switch associated with the EP device across the domain, the hardware implementation method of HPA to DPA mapping is used to complete the address translation on the cross-domain path; at the same time, the embodiment of the present application also uses FM to uniformly manage the allocation of segment numbers of all host domains in the topology, associates the segment number with the host domain, and the segment numbers between different hosts do not overlap; implements the segment / t / PID mapping table on the DSP of the destination switch, obtains the destination identifier, and uses the PBR routing method to complete the correct forwarding of the UIO completion message to the source vUSP.

[0073] The following uses Figure 1 and Figure 2 The topological structure shown exemplarily describes the relevant implementation details of cross-domain path management in the embodiment of the present application.

[0074] 1. When the downstream edge switch is initialized, the status of all EP devices that need to cross domains is set to unbound, that is, they are not visible to the hosts in the host domain where the EP devices are located. Figure 1 and Figure 2 The EP devices EP0, EP1, EP2 and EP3 shown are all devices that need to cross domains. It can be set that EP0, EP1, EP2 are invisible to the host Host0, and EP3 is invisible to Host1.

[0075] 2. FM can use the crawl out command to discover the entire topology, and use the PID allocation command to complete the PID allocation of all devices in the topology. Figure 1 , Figure 2 and Figure 3 As shown, the PIDs of the upstream edge ports vUSP of switch0 and Switch1 are 0 and 1 respectively. The PIDs of the downstream edge ports DSP connected to EP0, EP1 and EP3 are 8, 5 and 7 respectively.

[0076] 3. FM uses user-defined commands to read and write the configuration space of the PCIe device through the CCI interface of the switch using the CfgRd0 / CfgWr0 command:

[0077] Read and write the Bar register of the PCIe device to obtain the DPA space size of the EP device.

[0078] User-defined Opcodes are C000h-FFFFh.

[0079] The specific contents of the user-defined commands in this application are shown in the following table:

[0080]

[0081]

[0082] In the table, the register number is set to 10h, 14h, 18h or 1ch, which means that by reading and writing the address, the size of the memory space applied for each Bar register space of the EP device can be obtained, that is, the DPA size.

[0083] 4. After FM collects information about all EP devices, it exchanges information with the hosts in the host domain. Figure 1 , Figure 2 and Figure 3, inform Host0 of the size of Host1.EP1's DPA space, and inform Host1 of the size of Host0.EP0's DPA space. The host will allocate a cross-domain address space (GIM) within the address range of fabric_base[51:36] and fabric_limit[51:36] based on the size of the cross-domain device's DPA space, and the address space can be accessed using UIO messages. At the same time, the host will configure the FAST routing table in the host edge switch and / or downstream edge switch. This table is the routing relationship between HPA and DPID, and between DPID and egress port. This process complies with the CXL protocol. Figure 3 As shown, for cross-domain path 1, a FAST routing table is configured on the host edge switch Switch1, for cross-domain path 2, a FAST routing table is configured on the host edge switch Switch0, for cross-domain path 3, a FAST routing table is configured on the downstream edge switch Switch2, and the FAST routing table is configured by the host in the domain where the switch is located.

[0084] 5. FM assigns segment numbers to hosts in the host domain. The segment numbers between host domains do not overlap. The purpose of this is to ensure that the UIO completion message returned by the EP device can be correctly routed to each host.

[0085] The details of using HPA / DPA conversion logic and segment / PID mapping table to complete address translation and routing information conversion on cross-domain paths are as follows. Figure 4 As shown:

[0086] 1. The CCI interface in the host edge switch and the downstream edge switch can receive user-defined commands from the FM and parse them. In the user-defined command, the outbound port number of the switch at this level (port ID in the table) and the offset address of the configuration space register of each EP device have been specified. The hardware CCI interface can convert the user-defined command into the CfgWr0 / CfgRd0 command to access the configuration space of the EP device. The process can be found in the relevant description of the user-defined command above, which will not be repeated here.

[0087] 2. At the DSP position of the downstream edge switch, implement the HPA2DPA mapping table. Among them, the HPA address HPA[51:36] has a mapping relationship with the DPID, which is reflected in the FAST table. HPA[35:0] is the offset address (offset) of the memory space of the EP device.

[0088] At the DSP of the downstream edge switch, the hardware detection logic for writing the Bar register of the EP device is implemented. When the DSP forwards CfgWr0 (Bar register of Ep3) (indicating a write operation to the Bar register of Ep3), if it is detected that it is a write to the Bar register, the value to be written is latched as the base address of the DPA of the EP device.

[0089] 3. After the UIO request message with DPID is forwarded to the DSP of the downstream edge switch, the DPID information is stripped, and the HPA is mapped to the DPA (that is, the base address of the DPA + offset), completing the conversion from the PBR packet to the HBR (hierarchy based routing) packet.

[0090] 4. A segment / PID table will also be established at the DSP of the downstream edge switch. The DSP will obtain the SPID in the UIO request message and store the association between the SPID and the segment in the UIO request message in the segment / PID table.

[0091] After the downstream edge switch transmits the HBR packet to the EP device through the DSP, the EP device generates a UIO completion message for the HBR packet. The UIO completion message carries the destination segment (the segment number of the destination, i.e., the segment number of the host to be returned).

[0092] After the downstream edge switch receives the UIO completion message from the EP device through the DSP, it will first look up the segment / PID table. If it hits the segment / PID table, it will obtain the corresponding PID from the table based on the destination segment. This PID can be used as the SPID to group PBR packets. The PBR packets with the group number are then forwarded to the source host (that is, the host corresponding to the SPID) to complete cross-domain access.

[0093] This application manages the entire topology in a unified manner based on FM, obtains the DPA space size of the cross-domain device according to the user-defined command, and prepares for the host to establish the GIM address space. PBR routing is used on the cross-domain path, and cross-domain is no longer limited to the tree topology of PCIe, and cross-domain can be realized in more complex topologies.

[0094] On the cross-domain device side, the DSP above the device implements the segment / PID mapping table, the cross-domain address mapping relationship conversion logic, and the UIO request message processing logic, which is used to strip the DPID information in the UIO request message (the purpose is to convert the PBR packet into the HBR packet) and replace the HPA. The processing logic is simple to implement and does not introduce too much overhead in terms of resources and area.

[0095] This application is based on the CXL protocol and the PBR routing method. On the cross-domain path, the cross-domain path and the non-cross-domain path can share the physical port (physical path), and there is no need to allocate a dedicated physical port for the cross-domain path.

[0096] Accordingly, the present application provides a cross-domain access method in a second aspect.

[0097] The cross-domain access method is implemented in a multi-host system, the multi-host system comprising: a fabric manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the fabric manager is used to discover and manage the topology of the multi-host system, and allocate segment numbers to the hosts in the host domain and port identification numbers to the host edge switch, the downstream edge switch, and the terminal device based on the topology; the fabric manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains; the cross-domain access method comprises:

[0098] During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device that is accessed across the domain through the associated host edge switch via the port-based routing switching network;

[0099] After receiving the request message, the downstream edge switch associated with the terminal device accessed across domains establishes an association relationship between the segment sequence number and the port identification number carried in the request message, and maps the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwards it to the terminal device accessed across domains;

[0100] The downstream edge switch associated with the terminal device accessed across domains receives the completion message returned by the terminal device accessed across domains, and determines the port identification number corresponding to the segment number carried in the completion message based on the association relationship. Then, the port identification number corresponding to the segment number carried in the completion message is carried as the destination identifier in the completion message, and is sent to the corresponding host by the port-based routing switching network.

[0101] In some optional implementations,

[0102] The device information includes the DPA space size of the terminal device; the method further includes:

[0103] The fabric manager sends a custom command to the component command interface of the downstream edge switch to obtain the DPA space size of the terminal device;

[0104] The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message;

[0105] The step of mapping the cross-domain host physical address in the request message to the cross-domain host physical address and then forwarding the physical address to the terminal device accessed across the domain includes:

[0106] The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device being accessed cross-domain based on the base address.

[0107] In some optional implementations, the method further includes:

[0108] The host that requires cross-domain access is also used to configure a routing table for the host edge switch that requires cross-domain access, and the routing table includes a first mapping relationship between the cross-domain host physical address of the host that requires cross-domain access and the port identification number of the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host that requires cross-domain access.

[0109] In some optional implementations, the first mapping relationship is a mapping relationship between a high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is an offset address of the terminal device corresponding to the port identification number.

[0110] In some optional implementations, mapping the physical address of the host to be accessed across domains in the request message to the physical address of the host to be accessed across domains, and then forwarding it to the terminal device to be accessed across domains, includes:

[0111] The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the host physical address required to be accessed across domains carried in the request message with the base address of the host physical address of the terminal device being accessed across domains, thereby realizing address mapping from the host physical address required to be accessed across domains to the physical address to be accessed across domains.

[0112] In some optional implementations, the request message carries the cross-domain host physical address of the host to be accessed across domains and the port identification number of the edge port connected to the terminal device to be accessed across domains; the method further includes:

[0113] After receiving the request message from the host, the host edge switch determines the edge port through the routing table, and transmits the request message from the edge port to the port-based routing switching network.

[0114] In some optional implementations, the method further includes:

[0115] When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

[0116] The above method is implemented in the multi-host system provided in the above embodiment. The specific implementation method can be found in the description of the embodiment of the multi-host system above, and will not be repeated here.

[0117] It is understood that the circuit structures, names and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above embodiments according to the use requirements, and should not limit the concept of the present application to the specific details of the above examples.

[0118] Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that he or she may still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-host system, characterized in that: include: A fabric manager and at least two host domains; the host domains comprising a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; The structure manager is used to discover and manage the topology of the multi-host system, and allocate segment numbers to hosts in the host domain based on the topology, and allocate port identification numbers to the host edge switch, the downstream edge switch and the terminal device; the structure manager is also used to obtain device information of the terminal device, and send the device information to the host that needs to access the terminal device across domains; During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device to be accessed across the domain through the host edge switch and the port-based routing switching network; After receiving the request message, the downstream edge switch associated with the terminal device that is accessed across domains establishes an association relationship between the segment number and the port identification number carried in the request message, and maps the physical address of the host that needs to cross domains in the request message to the physical address of the host that is accessed across domains, and then forwards it to the terminal device that is accessed across domains; the downstream edge switch associated with the terminal device that is accessed across domains also receives a completion message returned by the terminal device that is accessed across domains, and after determining the port identification number corresponding to the segment number carried in the completion message based on the association relationship, carries the port identification number corresponding to the segment number carried in the completion message as the destination identifier in the completion message, and sends it to the corresponding host by the port-based routing switching network.

2. The multi-host system according to claim 1, characterized in that: The device information includes the DPA space size of the terminal device; the fabric manager uses a custom command to obtain the DPA space size of the terminal device through the component command interface of the downstream edge switch; The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message; The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device accessed across domains based on the base address.

3. The multi-host system according to claim 1 or 2, characterized in that: The host that requires cross-domain access is also used to configure a routing table for the host edge switch that requires cross-domain access, and the routing table includes a first mapping relationship between the cross-domain host physical address of the host that requires cross-domain access and the port identification number of the edge port connected to the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host that requires cross-domain access.

4. The multi-host system according to claim 3, characterized in that: The first mapping relationship is the mapping relationship between the high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is the offset address of the terminal device corresponding to the port identification number.

5. The multi-host system according to claim 4, characterized in that: The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the physical address of the host required to cross domains carried in the request message with the base address of the physical address of the host being accessed across domains of the terminal device being accessed across domains, thereby realizing address mapping from the physical address of the host required to cross domains to the physical address of the host being accessed across domains.

6. The multi-host system according to claim 3, characterized in that: The request message carries the physical address of the host that needs cross-domain access and the port identification number of the terminal device that is accessed across domains; after the host edge switch receives the request message from the host, it determines the outbound port through the routing table, and transmits the request message from the outbound port to the port-based routing switching network.

7. The multi-host system according to any one of claims 1, 2, 4-6, characterized in that: When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

8. A cross-domain access method, characterized in that: The cross-domain access method is implemented in a multi-host system, the multi-host system comprising: a fabric manager and at least two host domains; the host domain comprises a host, a host edge switch associated with the host, at least one terminal device, and a downstream edge switch associated with the terminal device; the fabric manager is used to discover and manage the topology of the multi-host system, and allocate segment numbers to the hosts in the host domain and port identification numbers to the host edge switch, the downstream edge switch, and the terminal device based on the topology; the fabric manager is also used to obtain device information of the terminal device and send the device information to the host that needs to access the terminal device across domains; the cross-domain access method comprises: During the cross-domain access process, the host that needs cross-domain access sends a request message to the terminal device that is accessed across the domain through the associated host edge switch via the port-based routing switching network; After receiving the request message, the downstream edge switch associated with the terminal device accessed across domains establishes an association relationship between the segment sequence number and the port identification number carried in the request message, and maps the physical address of the host requiring cross-domain access in the request message to the physical address of the host being accessed across domains, and then forwards it to the terminal device accessed across domains; The downstream edge switch associated with the terminal device accessed across domains receives the completion message returned by the terminal device accessed across domains, and determines the port identification number corresponding to the segment number carried in the completion message based on the association relationship. Then, the port identification number corresponding to the segment number carried in the completion message is carried as the destination identifier in the completion message, and is sent to the corresponding host by the port-based routing switching network.

9. The cross-domain access method according to claim 8, characterized in that: The device information includes the DPA space size of the terminal device; the method further includes: The fabric manager sends a custom command to the component command interface of the downstream edge switch to obtain the DPA space size of the terminal device; The edge port on the downstream edge switch connected to the terminal device captures the configuration message of the DPA of the terminal device, and latches the base address of the cross-domain host physical address of the DPA of the terminal device carried in the configuration message; The mapping of the cross-domain host physical address in the request message to the cross-domain host physical address and then forwarding the physical address to the terminal device accessed across the domain includes: The edge port of the downstream edge switch maps the cross-domain host physical address carried in the request message to the cross-domain host physical address of the terminal device accessed across domains based on the base address.

10. The cross-domain access method according to claim 8 or 9, characterized in that: The method further comprises: The host that requires cross-domain access is also used to configure a routing table for the host edge switch that requires cross-domain access, and the routing table includes a first mapping relationship between the cross-domain host physical address of the host that requires cross-domain access and the port identification number of the terminal device that can be accessed across the domain, and a second mapping relationship between the port identification number and the outbound port of the host edge switch associated with the host that requires cross-domain access.

11. The cross-domain access method according to claim 10, characterized in that: The first mapping relationship is the mapping relationship between the high-order address of the host physical address that needs to cross domains and the port identification number; the low-order address of the host physical address that needs to cross domains is the offset address of the terminal device corresponding to the port identification number.

12. The cross-domain access method according to claim 11, characterized in that: The mapping of the cross-domain host physical address in the request message to the cross-domain host physical address and then forwarding the physical address to the terminal device accessed across the domain includes: The downstream edge switch associated with the terminal device being accessed across domains replaces the high-order address of the host physical address required to be accessed across domains carried in the request message with the base address of the host physical address of the terminal device being accessed across domains, thereby realizing address mapping from the host physical address required to be accessed across domains to the physical address to be accessed across domains.

13. The cross-domain access method according to claim 11, characterized in that: The request message carries the physical address of the host that needs to be accessed across domains and the port identification number of the edge port connected to the terminal device that is accessed across domains; the method further includes: After receiving the request message from the host, the host edge switch determines the outbound port through the routing table, and transmits the request message from the edge port to the port-based routing switching network.

14. The cross-domain access method according to any one of claims 8, 9, 11-13, characterized in that: The method further comprises: When the downstream edge switch is initialized, the terminal device to be accessed across domains is configured to be invisible to the host across domains.

Images