Routing implementation method and apparatus for pcie topological structure between dies

Abstract

Provided in the present disclosure are a routing implementation method and apparatus for a PCIe topological structure between dies. The method comprises: on the basis of the number of dies in a PCIe switch, dividing an original routing table into a plurality of routing sub-tables, and respectively storing same in corresponding dies, wherein each routing sub-table only comprises routing information of ports in a corresponding die; after a message is received at an ingress port of a first die, using a routing sub-table of the first die to match an egress port for the message, and if no egress port is matched, using a transmission port between dies to forward the message to a second die, and using a routing sub-table of the second die to continue matching, and if no egress port is matched, continuing forwarding the message until an egress port is successfully matched, and sending the message; and if the message has been forwarded to all dies in the PCIe switch and still no egress port is matched, outputting the message to the ingress port of the first die, and also reporting error processing information. The technical solution of the present disclosure reduces the resource storage amount of a routing table, avoids chip wiring congestion, and reduces the difficulty of chip design.

Description

A routing implementation method and device for a PCIe topology structure between Dies

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202410330167.8 filed on March 21, 2024, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0003] The present disclosure belongs to the field of data routing, and in particular relates to a method and device for implementing routing of an inter-Die PCIe topology structure. Background Art

[0004] A PCIe switch (an expansion device in the PCIe topology that can be used to expand the number of PCIe channels, thereby enhancing the expansion capabilities of the main CPU's PCIe devices) is an expansion device in the PCIe (PCI-Express, peripheral component interconnect express, a high-speed serial computer expansion bus standard) topology. It is widely used in card devices in large storage systems or server platforms to expand the number of PCIe lanes and facilitate data exchange between PCIe channels, thereby enhancing the expansion capabilities of the main CPU's PCIe devices. As the number of PCIe lanes increases, the exchange between channels becomes increasingly complex, posing a significant challenge to physical implementation. Therefore, within the limitations of existing physical specifications, a die-to-die approach (interconnection between chips) can be used to distribute PCIe channels across multiple dies (unpackaged chips) to reduce the physical implementation complexity of a single die. However, routing information is also distributed across multiple dies, making communication between dies a key issue.

[0005] The PCIe protocol or traditional implementation methods all aggregate the BDF information (including bus number, device number, and function number) or BAR (base address register) address information of all ports into a unified routing table, which is then routed once to obtain the result. If there are no multiple dies, single-routing is the simplest and most convenient solution. For multi-die PCIe switch chips, if single-routing is still used, the routing information of all ports between multiple dies must be synchronized, and a new protocol must be defined between dies to transmit this routing information. This not only increases complexity but also reduces the effective bandwidth between dies. In addition, each die needs to store a complete routing table, which increases the area of ​​the routing table and makes back-end timing convergence difficult at high clock frequencies. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a method and device for implementing routing of a PCIe topology structure between Dies, aiming to solve the PCIe routing problem between Dies through a multiple routing solution.

[0007] According to a first aspect of the present disclosure, a method for implementing routing of an inter-Die PCIe topology structure is provided, comprising:

[0008] According to the number of dies in the PCIe switch, the original routing table is divided into multiple routing sub-tables and stored in the corresponding dies. Each routing sub-table only includes the routing information of the ports in the corresponding die.

[0009] When a message is received at the ingress port of the first Die, the routing sub-table of the first Die is used to match the message's egress port. If no egress port is matched, the message is forwarded to the second Die using the inter-Die transmission port. The second Die's routing sub-table is used to continue matching. If no match is found, the message is forwarded again until the egress port is successfully matched and the message is sent out.

[0010] If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error has occurred, the message is returned to the ingress port of the first Die, and error handling information is reported.

[0011] Optionally, each Die includes the inter-Die transmission port, the inter-Die transmission port includes an inter-Die input port and an inter-Die output port, and the first Die forwards the message to the second Die, further comprising:

[0012] The message is sent to the second Die through the Die output port of the first Die. e Di e Time input port.

[0013] Optionally, the routing subtable includes multiple port entries and unmatched entries, and the routing subtable using the second Die continues to match, and continues to forward if there is no match, further comprising:

[0014] If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

[0015] Optionally, the number of routing failures is recorded in the descriptor of the message, and when no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

[0016] Optionally, determining that a message error occurs further includes:

[0017] When the number of routing failures is equal to the number of dies in the PCIe switch, it is determined that the message has been forwarded to all dies in the PCIe switch but still has not been matched to an egress port, and thus a message error is determined to have occurred.

[0018] According to a second aspect of the present disclosure, a device for implementing routing of an inter-Die PCIe topology structure is provided, comprising:

[0019] The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively. Each routing sub-table only includes the routing information of the ports in the corresponding die;

[0020] An inter-Die routing unit, configured to, upon receiving a message at the ingress port of a first Die, match the message's egress port using the routing sub-table of the first Die. If no egress port is matched, forward the message to the second Die using the inter-Die transmission port, continue matching using the routing sub-table of the second Die, and continue forwarding if no match is found until the egress port is successfully matched, and then send the message;

[0021] The error handling unit is configured to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, return the message to the ingress port of the first Die, and report error handling information.

[0022] The technical solution disclosed in this disclosure has at least the following advantages:

[0023] Because the present disclosure only stores the current die's information on a die, the routing table resources only require 1 / N of the total port routing information (N is the number of dies). This reduces routing table resource storage, avoids chip wiring congestion, and eliminates the need to design a separate mechanism for synchronizing routing information between dies, reducing design difficulty. As the number of channels increases, it can be expanded to scenarios with more die interconnections, such as 8-die and 16-die routing.

[0024] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be achieved and obtained through the structures and processes indicated in the description and the drawings.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. It is obvious that the drawings described below are certain embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] FIG1 is a flow chart of a method for implementing routing of an inter-Die PCIe topology structure according to an embodiment of the present disclosure.

[0028] FIG2 is a schematic diagram of a successful one-time routing path of a PCIe Switch according to an embodiment of the present disclosure.

[0029] FIG3 is a schematic diagram of multiple successful routing paths of a PCIe Switch according to an embodiment of the present disclosure.

[0030] FIG4 is a schematic diagram of a PCIe Switch routing failure path according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present disclosure.

[0032] Based on the above analysis, this disclosure proposes a method and device for implementing routing in an inter-die PCIe topology. To address the issue of large routing tables, the routing table is split into separate dies based on the die partitioning method. Each die's routing sub-table only retains the port information on the current die. When a message enters each die from the ingress port, only the routing information on the current die is queried. If routing fails on a die, the message is sent to the next die for continued routing until it is finally routed to the egress port.

[0033] 1 , illustratively, the routing implementation method of the inter-die PCIe topology structure provided by the present disclosure includes:

[0034] Step 101: Divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively. Each routing sub-table only includes routing information of the ports in the corresponding die.

[0035] For example, referring to Figure 2, the present disclosure is described using a PCIe switch with 16 ports and 4 Die interconnects as an example. Those skilled in the art will appreciate that the structure shown in Figure 2 is merely for illustrating the principles of the present disclosure and is not limited to the above specific example.

[0036] The PCIe switch consists of a routing table consisting of routing information of 16 ports. The present disclosure divides the entire routing table into four parts according to the situation of the ports on the Die, and only stores the routing information of all ports on the current Die on Die0, Die1, Die2 and Die3, that is, Die0 stores the routing information of port0 to port3, Die1 stores the routing information of port4 to port7; Die2 stores the routing information of port8 to port11, and Die3 stores the routing information of port12 to port15. As an optional implementation method, the present disclosure defines an inter-Die transmission port D2D_PORT, including an input port D2D_PORT_IN and an output port D2D_PORT_OUT, as a data transmission channel between Dies, and four Dies are connected into a ring network through D2D_PORT. As an optional implementation method, the present disclosure defines a message descriptor:

[0037] src_port_id: The inbound port number of the message, indicating the physical port from which the message enters;

[0038] nomatch_cnt: The number of routing failures. The initial value is 0. It will increase by 1 after each Die routing failure. The maximum value does not exceed the number of Dies.

[0039] Step 102: When a message is received at the ingress port of the first Die, the routing sub-table of the first Die is used to match the egress port of the message. If no egress port is matched, the message is forwarded to the second Die using the inter-Die transmission port. The second Die's routing sub-table is used to continue matching. If no match is found, the message is forwarded again until the egress port is successfully matched and the message is sent.

[0040] Based on the above structure and definition, this disclosure describes inter-Die routing based on three typical scenarios.

[0041] For example, Figure 2 illustrates the first scenario. After the message passes through the current Die's routing subtable, it finds the outgoing port and is then routed out of the current Die's port. For example, in the four Dies shown in Figure 2, port 0 to port 1, port 6 to port 7, port 9 to port 10, and port 12 to port 13 all pass through the current Die's routing subtable before being routed out of the current Die. In this scenario, D2D_PORT is not required.

[0042] For example, Figure 3 shows the second scenario. The message needs to be routed through multiple dies before it finds the egress port, and the message is routed out from other dies. Figure 3 describes the entire data path for a message entering from port 0, routing through four dies, and finally being routed out from port 13. The specific steps are as follows:

[0043] The message enters from Port 0 of Die 0 and is queried through the routing subtable (ROUTE_TABLE). If no match is found for any port on Die 0 (NO_MATCH), the message is added with the descriptor information {(src_port_id=0), (no_match_cnt=1)} and sent to the inter-die output port D2D_PORT_OUT; as shown in steps 1 to 3 in Figure 3;

[0044] Die1's inter-die input port D2D_PORT_IN receives the message, checks if no_match_cnt < 4, and sends it to the routing subtable for a second routing. If no match is found for any port on Die1 (NO_MATCH), the descriptor information is modified to (no_match_cnt = 2) and sent to D2D_PORT_OUT; as shown in steps 4 to 6 in Figure 3;

[0045] Die2 repeats the operation of Die1 and modifies the descriptor information to (no_match_cnt == 3); this corresponds to steps 7 to 9 in FIG3;

[0046] The message enters from D2D_PORT_IN of Die3. At this time, no_match_cnt=3, and the normal routing table lookup is still performed. If port 13 is found, the message is sent from port 13. Figure 3 shows steps 10 to 11.

[0047] Step 103: If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error has occurred, and the message is returned to the ingress port of the first Die, while reporting error handling information.

[0048] For example, Figure 4 shows the third scenario. After the message passes through all Die routes, it still cannot find the egress port. The current Die needs to generate an abnormal message and output it from the ingress port of the current Die. The specific steps are as follows:

[0049] If the message enters from Port 1 of Die0, and the routing subtable (ROUTE_TABLE) is queried and the result does not match any port on Die0 (NO_MATCH), then the descriptor information {(src_port_id=1), (no_match_cnt=1)} is added to the message and sent to D2D_PORT_OUT; as shown in steps 1 to 3 in Figure 4;

[0050] When Die1's D2D_PORT_IN receives a message, it checks if no_match_cnt < 4 and sends it to the routing subtable for a second routing. If no port on Die1 is matched (NO_MATCH), the descriptor information in the message is modified to {(src_port_id = 1, (no_match_cnt = 2)} and sent to D2D_PORT_OUT; this corresponds to steps 4 to 6 in Figure 4;

[0051] Die2 and Die3 repeat the operation of Die1; corresponding to steps 7 to 12 in Figure 4;

[0052] The message enters Die0's D2D_PORT_IN. At this time, no_match_cnt == 4, indicating that the message still cannot find an exit after passing through all Die routes. The message then enters the ROUTE_ERR_PRO error handling module for error processing. The error is reported back to port1 based on src_port_id. This corresponds to steps 13-14 in Figure 4.

[0053] In summary, the present disclosure can cover multiple scenarios including single-die routing, multi-die routing and routing failure through a multi-die routing solution, and can cover all routing scenarios while reducing routing table resources.

[0054] It can be seen that the routing implementation method of the inter-die PCIe topology structure proposed in this disclosure has the following advantages compared to related technologies:

[0055] Because the present disclosure only stores the current die's information on a die, the routing subtable only uses 1 / N of the total port routing information (N is the number of dies). This reduces routing table storage resources, avoids chip wiring congestion, and eliminates the need for a separate mechanism to synchronize routing information between dies, reducing design complexity. As the number of channels increases, this approach can be expanded to scenarios with more die interconnections, such as 8-die and 16-die routing.

[0056] Accordingly, in a second aspect, the present disclosure provides a routing implementation device for an inter-Die PCIe topology structure, including:

[0057] The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively. Each routing sub-table only includes the routing information of the ports in the corresponding die;

[0058] The inter-Die routing unit is configured to, upon receiving a message at the ingress port of the first Die, match the message's egress port using the routing sub-table of the first Die. If no egress port is matched, forward the message to the second Die using the inter-Die transmission port, continue matching using the routing sub-table of the second Die, and continue forwarding if no match is found until the egress port is successfully matched and the message is sent out;

[0059] The error handling unit is used to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but has not yet matched an egress port, return the message to the ingress port of the first Die, and report error handling information.

[0060] The above-mentioned device can be implemented by the routing implementation method of the inter-Die PCIe topology structure provided by the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect and will not be repeated here.

[0061] It is understood that the circuit structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art may also readily conceive of combinations and adjustments to the structural features of the above embodiments as needed, and should not limit the concepts of the present disclosure to the specific details of the above examples.

[0062] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A routing implementation method for a Die-to-Die PCIe topology structure, characterized in that: include: According to the number of dies in the PCIe switch, the original routing table is divided into multiple routing sub-tables and stored in the corresponding dies. Each routing sub-table only includes the routing information of the ports in the corresponding die. When a message is received at the ingress port of the first Die, the routing sub-table of the first Die is used to match the message's egress port. If no egress port is matched, the message is forwarded to the second Die using the inter-Die transmission port. The second Die's routing sub-table is used to continue matching. If no match is found, the message is forwarded again until the egress port is successfully matched and the message is sent out. If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error has occurred, the message is returned to the ingress port of the first Die, and error handling information is reported.

2. The routing implementation method of the inter-die PCIe topology structure according to claim 1, characterized in that: Each Die includes the inter-Die transmission port, and the inter-Die transmission port includes an inter-Die input port and an inter-Die output port. The first Die forwards the message to the second Die, further comprising: The message is sent to the inter-Die input port of the second Die through the inter-Die output port of the first Die.

3. The routing implementation method of the inter-die PCIe topology structure according to claim 2, characterized in that: The routing sub-table includes multiple port entries and unmatched entries, and the routing sub-table using the second Die continues to match, and continues to forward if there is no match, further including: If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

4. The routing implementation method of the inter-die PCIe topology structure according to claim 1, characterized in that: The number of routing failures is recorded in the descriptor of the message. When no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

5. The routing implementation method of the inter-die PCIe topology structure according to claim 4, characterized in that: The determining that a message error occurs further includes: When the number of routing failures is equal to the number of dies in the PCIe switch, it is determined that the message has been forwarded to all dies in the PCIe switch but still has not been matched to an egress port, and thus a message error is determined to have occurred.

6. A routing implementation device for a Die-to-Die PCIe topology structure, characterized in that: include: The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively. Each routing sub-table only includes the routing information of the ports in the corresponding die; An inter-Die routing unit, configured to, upon receiving a message at the ingress port of a first Die, match the message's egress port using the routing sub-table of the first Die. If no egress port is matched, forward the message to the second Die using the inter-Die transmission port, continue matching using the routing sub-table of the second Die, and continue forwarding if no match is found until the egress port is successfully matched, and then send the message; The error handling unit is configured to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, return the message to the ingress port of the first Die, and report error handling information.

7. The routing implementation device for the inter-die PCIe topology structure according to claim 6, characterized in that: Each Die includes the inter-Die transmission port, and the inter-Die transmission port includes an inter-Die input port and an inter-Die output port. The inter-Die routing unit is further configured to: The message is sent to the inter-Die input port of the second Die through the inter-Die output port of the first Die.

8. The routing implementation device for the inter-die PCIe topology structure according to claim 7, characterized in that: The routing sub-table includes multiple port entries and unmatched entries, and the inter-die routing unit is further configured to: If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

9. The routing implementation device for an inter-die PCIe topology structure according to claim 6, characterized in that: The number of routing failures is recorded in the descriptor of the message. When no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

10. The routing implementation device for the inter-die PCIe topology structure according to claim 9, characterized in that: The error handling unit is further configured to: When the number of routing failures is equal to the number of dies in the PCIe switch, it is determined that the message has been forwarded to all dies in the PCIe switch but still has not been matched to an egress port, and thus a message error is determined to have occurred.

Images