A method and device for implementing routing of PCIe topology structure between Dies

By dividing the routing table into intra-die routing subtables and forwarding packets one by one, the complexity and bandwidth problems of inter-die PCIe routing are solved, resource conservation and design simplification are achieved, and it is suitable for more Die interconnection scenarios.

CN118171626BActive Publication Date: 2024-11-15WUXI STARS MICRO SYSTEM TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410330167.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-21
Publication Date
2024-11-15
Estimated Expiration
2044-03-21

AI Technical Summary

Technical Problem

In multi-Die PCIe switch chips, it is difficult for the prior art to effectively realize PCIe routing between Die and Die, resulting in complex synchronization of routing information, reduced bandwidth and large routing table occupancy, especially at high clock frequency, back-end timing convergence is difficult.

Method used

Divide the original routing table into multiple routing subtables. Each Die only stores the routing information of its internal ports, and forwards the packets divide through the inter-diie transmission port until the outgoing port is matched or the message error is determined.

Benefits of technology

Reduces routing table resource usage, avoids chip routing congestion, reduces design difficulty, and supports more Die interconnection scenarios, such as 8Die or 16Die routing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118171626B_ABST
    Figure CN118171626B_ABST
Patent Text Reader

Abstract

The present invention provides a routing implementation method and device for a PCIe topology structure between Dies, the method comprising: according to the number of Dies in a PCIe switch, dividing an original routing table into a plurality of routing sub-tables and storing them in corresponding Dies respectively, each routing sub-table only including routing information of a port in a corresponding Die; when a message is received at an input port of a first Die, matching an output port of the message using the routing sub-table of the first Die, if the output port is not matched, forwarding the message to a second Die using an inter-Die transmission port, continuing matching using the routing sub-table of the second Die, if not matched, continuing forwarding until the output port is successfully matched and sending the message; if the message has been forwarded to all Dies in a PCIe switch and still not matched to an output port, outputting the message to the input port of the first Die, and reporting error handling information at the same time. The technical solution of the present invention reduces the storage capacity of routing table resources, avoids chip wiring congestion, and reduces the difficulty of chip design.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data routing, and in particular relates to a routing implementation method and device for a PCIe topology structure between Dies. Background Art

[0002] PCIe Switch is an expansion device in the PCIe topology. It is widely used in board devices of large storage systems or server platforms. It is used to expand the number of PCIe channels (lanes) and complete data exchange between PCIe channels, thereby enhancing the expansion capabilities of the PCIe devices of the main CPU. As the number of PCIe channels increases, the exchange between channels becomes more and more complex, posing a major challenge to physical implementation. Therefore, under the limitations of existing physical specifications, the PCIe channels can be distributed to multiple Dies through the Die2Die (interconnection between core particles) method, reducing the difficulty of physical implementation of a single Die, but routing information is also distributed to multiple Dies, so how Dies communicate with each other becomes a key issue.

[0003] The PCIe protocol or traditional implementation method is to collect the BDF information (including bus number, device number and function number) or BAR (base address register) address information of all ports, unify them into a routing table, and perform one routing to get the result. If there are no multiple dies, one-time routing is the simplest and most convenient solution. For multi-die PCIe switch chips, if one-time routing is still used, the routing information of all ports between multiple dies must be synchronized, and new protocols must be defined between Die2Die to transmit this routing information, which not only increases the complexity, but also reduces the effective bandwidth between Die2Die. In addition, each Die needs to store a complete routing table, which increases the area of ​​the routing table, and the back-end timing convergence is difficult when the clock frequency is high. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for implementing routing of a PCIe topology structure between Dies, aiming to solve the routing problem of PCIe between Dies through multiple routing schemes.

[0005] According to a first aspect of the present invention, a method for implementing routing of a PCIe topology structure between Dies is provided, comprising:

[0006] According to the number of dies in the PCIe switch, the original routing table is divided into multiple routing sub-tables and stored in the corresponding dies respectively. Each routing sub-table only includes the routing information of the port in the corresponding die.

[0007] When a message is received at the inbound port of the first Die, the outbound port of the message is matched using the routing subtable of the first Die. If no outbound port is matched, the message is forwarded to the second Die using the inter-Die transmission port, and the second Die routing subtable is used to continue matching. If no match is found, the message is forwarded until the outbound port is matched successfully and the message is sent out;

[0008] If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error occurs, and the message is returned to the ingress port of the first Die, while reporting error processing information.

[0009] Preferably, each Die includes the inter-Die transmission port, the inter-Die transmission port includes an inter-Die input port and an inter-Die output port, and the first Die forwards the message to the second Die, further comprising:

[0010] The message is sent to the inter-Die input port of the second Die through the inter-Die output port of the first Die.

[0011] Preferably, the routing subtable includes multiple port items and unmatched items, and the routing subtable using the second Die continues to match, and continues to forward if there is no match, further comprising:

[0012] If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

[0013] Preferably, the number of routing failures is recorded in the descriptor of the message, and when no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

[0014] Preferably, the determining that a message error occurs further comprises:

[0015] When the number of routing failures is equal to the number of dies in the PCIe switch, it is determined that the message has been forwarded to all dies in the PCIe switch but still has not been matched to an egress port, that is, it is determined that a message error occurs.

[0016] According to a second aspect of the present invention, a routing implementation device for a Die-to-Die PCIe topology structure is provided, comprising:

[0017] The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively, and each routing sub-table only includes the routing information of the port in the corresponding die;

[0018] An inter-Die routing unit is used to match the egress port of the message using the routing sub-table of the first Die when receiving the message at the ingress port of the first Die. If the egress port is not matched, the message is forwarded to the second Die using the inter-Die transmission port, and the matching is continued using the routing sub-table of the second Die. If there is no match, the message is forwarded until the egress port is successfully matched and the message is sent out;

[0019] The error processing unit is used to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, return the message to the ingress port of the first Die, and report error processing information.

[0020] Compared with the related art, the technical solution of the present invention has the following advantages:

[0021] Since the present invention only stores the information of the current Die on a Die, the resources of the routing table are only 1 / N (N is the number of Dies) of the routing information of all ports, thus reducing the storage capacity of the routing table resources, avoiding the problem of chip wiring congestion, and there is no need to design a separate synchronization mechanism for routing information between Dies, reducing the design difficulty. As the number of channels increases, it can be expanded to more Die interconnected scenarios, such as 8Die and 16Die routing.

[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the structures and processes indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. It is obvious that the drawings described below are certain embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flow chart of a method for implementing routing of a PCIe topology structure between Dies according to the present invention.

[0025] Figure 2This is a successful routing path diagram of the PCIe Switch according to the present invention.

[0026] Figure 3 It is a PCIe Switch multiple routing successful path diagram according to the present invention.

[0027] Figure 4 is a PCIe Switch routing failure path diagram according to the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Based on the above analysis, the present invention proposes a routing implementation method and device for the PCIe topology structure between Dies. In order to solve the problem of a large routing table, the routing table is split into respective Dies according to the Die division method, and the routing sub-table of each Die only retains the port information on the current Die. When the message enters each Die from the ingress port ingress, only the routing information on the current Die is queried. If the routing fails on a Die, the message is sent to the next Die to continue routing until it is finally routed to the egress port egress.

[0030] See also Figure 1 The routing implementation method of the inter-die PCIe topology structure provided by the present invention includes:

[0031] Step 101: According to the number of dies in the PCIe switch, the original routing table is divided into multiple routing sub-tables and stored in corresponding dies respectively, and each routing sub-table only includes routing information of the ports in the corresponding die.

[0032] See also Figure 2 The present invention is described by taking a PCIe switch with 16 ports and 4 Die interconnections as an example. Those skilled in the art can understand that Figure 2 The structures shown in the figures are only for illustrating the principles of the present invention and are not limited to the specific examples described above.

[0033] The PCIe switch consists of a routing table with routing information of 16 ports. The present invention divides the entire routing table into four parts according to the situation of the ports on the Die, and only stores the routing information of all ports on the current Die on Die0, Die1, Die2 and Die3, that is, Die0 stores the routing information of port0~port3, Die1 stores the routing information of port4~port7; Die2 stores the routing information of port8~port11, and Die3 stores the routing information of port12~port15. The present invention defines the inter-die transmission port D2D_PORT, including the input port D2D_PORT_IN and the output port D2D_PORT_OUT, as the data transmission channel between Dies, and the four Dies are connected into a ring network through D2D_PORT. The present invention defines the descriptor of the message:

[0034] src_port_id: the inbound port number of the message, indicating the physical port from which the message enters;

[0035] no_match_cnt: The number of routing failures. The initial value is 0. It will increase by 1 after each Die routing failure. The maximum value does not exceed the number of Dies.

[0036] Step 102: When a message is received at the input port of the first Die, the routing subtable of the first Die is used to match the output port of the message. If no output port is matched, the message is forwarded to the second Die using the inter-Die transmission port, and the matching is continued using the routing subtable of the second Die. If no match is found, the message is forwarded until the output port is matched successfully and the message is sent out.

[0037] Based on the above structure and definition, the present invention describes inter-Die routing based on three typical scenarios.

[0038] Figure 2 The first scenario is shown in Figure 1. After the message passes through the current Die routing subtable, it finds the outbound port and is routed out from the current Die port. Figure 2 Among the 4 Dies, port0 to port1, port6 ​​to port7, port9 to port10, and port12 to port13 are all situations where the message is output from the current Die after the routing subtable of the current Die is queried; at this time, D2D_PORT does not need to work.

[0039] Figure 3 The second scenario is shown in Figure 1. The message needs to pass through multiple Dies before finding the outbound port, and the message is routed from other Dies. Figure 3It describes the entire data path from the message entering from port 0, passing through 4 Die routes, and finally being routed out from port 13. The specific steps are as follows:

[0040] The message enters from Port0 of Die0, and is queried through the routing subtable (ROUTE_TABLE). If the result does not match any port on Die0 (NO_MATCH), then the descriptor information {(src_port_id=0), (no_match_cnt=1)} is added to the message and sent to the inter-die output port D2D_PORT_OUT; Figure 3 Steps 1 to 3 in the above code;

[0041] The Die input port D2D_PORT_IN of Diel receives the message, checks no_match_cnt<4, and sends it to the routing subtable for the second routing. If it does not match any port on Diel (NO_MATCH), the descriptor information is modified to (no_march_cnt=2) and sent to D2D_PORT_OUT; Figure 3 Steps 4 to 6 in

[0042] Die2 repeats the operation of Die1 and modifies the descriptor information to (no_match_cnt=3); Figure 3 Corresponding to steps 7 to 9;

[0043] The message enters from D2D_PORT_IN of Die3. At this time, no_match_cnt=3. The normal routing table lookup is still performed. If port 13 is hit, the message is sent from port 13. Figure 3 The middle is steps 10 to 11.

[0044] Step 103: If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error occurs, and the message is returned to the ingress port of the first Die, while reporting error processing information.

[0045] Figure 4 The third scenario is shown. After the message passes through all Die routes, it still cannot find the egress port. The current Die needs to generate an abnormal message and output it from the ingress port of the current Die. The specific steps are as follows:

[0046] If the message enters from Port 1 of Die0, and the result of querying the routing subtable (ROUTE_TABLE) does not match any port on Die0 (NO_MATCH), then the descriptor information {(src_port_id=1), (no_match_cnt=1)} is added to the message and sent to D2D_PORT_OUT; Figure 4 Steps 1 to 3 in the above code;

[0047] When D2D_PORT_IN of Die1 receives the message, it checks no_match_cnt<4, and sends it to the routing subtable for a second routing. If it does not match any port on Die1 (NO_MATCH), it modifies the descriptor information in the message to {(src_port_id=1), (no_match_cnt=2)} and sends it to D2D_PORT_OUT; Figure 4 Corresponding to steps 4 to 6;

[0048] Die2 and Die3 repeat the operation of Die1; Figure 4 Corresponding to steps 7 to 12;

[0049] The message enters from D2D_PORT_IN of Die0, and no_match_cnt=4 at this time; it means that the message still cannot find the exit after passing through all Die routes, so it enters the ROUTE_ERR_PRO error handling module for error handling, and reports the error according to src_port_id and returns it to port1. Figure 4 Corresponding to steps 13 to 14.

[0050] In summary, the present invention can cover multiple scenarios of single-Die routing, multi-Die routing and routing failure through a multi-Die routing solution, and can cover all routing scenarios while reducing routing table resources.

[0051] It can be seen that the routing implementation method of the inter-die PCIe topology structure proposed in the present invention has the following advantages compared with the related art:

[0052] Since the present invention only stores the information of the current Die on a Die, the resources of the routing subtable are only 1 / N (N is the number of Dies) of the routing information of all ports, thus reducing the storage capacity of the routing table resources, avoiding the chip wiring congestion problem, and there is no need to design a separate synchronization mechanism for routing information between Dies, reducing the design difficulty. As the number of channels increases, it can be expanded to more Die interconnected scenarios, such as 8Die and 16Die routing.

[0053] Accordingly, in a second aspect, the present invention provides a routing implementation device for a Die-to-Die PCIe topology structure, comprising:

[0054] The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively, and each routing sub-table only includes the routing information of the port in the corresponding die;

[0055] An inter-Die routing unit is used to match the egress port of the message using the routing sub-table of the first Die when receiving the message at the ingress port of the first Die. If the egress port is not matched, the message is forwarded to the second Die using the inter-Die transmission port, and the matching is continued using the routing sub-table of the second Die. If there is no match, the message is forwarded until the egress port is successfully matched and the message is sent out;

[0056] The error processing unit is used to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, return the message to the ingress port of the first Die, and report error processing information.

[0057] The above-mentioned device can be implemented by the routing implementation method of the Die PCIe topology structure provided by the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect, which will not be repeated here.

[0058] It is understood that the circuit structures, names and parameters described in the above embodiments are only examples. Those skilled in the art can also make easily conceivable combinations and adjustments to the structural features of the above embodiments according to the use requirements, and the concept of the present invention should not be limited to the specific details of the above examples.

[0059] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A routing implementation method for a PCIe topology structure between Dies, It is characterized in that include: According to the number of dies in the PCIe switch, the original routing table is divided into multiple routing sub-tables and stored in the corresponding dies respectively. Each routing sub-table only includes the routing information of the port in the corresponding die. When a message is received at the inbound port of the first Die, the outbound port of the message is matched using the routing subtable of the first Die. If no outbound port is matched, the message is forwarded to the second Die using the inter-Die transmission port, and the second Die routing subtable is used to continue matching. If no match is found, the message is forwarded until the outbound port is matched successfully and the message is sent out; If the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, it is determined that a message error occurs, and the message is returned to the ingress port of the first Die, while reporting error processing information.

2. The routing implementation method of the inter-die PCIe topology structure according to claim 1, It is characterized in that Each Die includes the inter-Die transmission port, the inter-Die transmission port includes an inter-Die input port and an inter-Die output port, and the first Die forwards the message to the second Die, further comprising: The message is sent to the inter-Die input port of the second Die through the inter-Die output port of the first Die.

3. The routing implementation method of the inter-die PCIe topology structure according to claim 2, It is characterized in that The routing subtable includes multiple port items and unmatched items, and the routing subtable using the second Die continues to match, and continues to forward if there is no match, further comprising: If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

4. The routing implementation method of the inter-die PCIe topology structure according to claim 1, It is characterized in that The number of routing failures is recorded in the descriptor of the message. When no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

5. The routing implementation method of the inter-die PCIe topology structure according to claim 4, It is characterized in that The determining that a message error occurs further includes: When the number of routing failures is equal to the number of Dies in the PCIe switch, it is determined that the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, that is, it is determined that a message error occurs.

6. A routing implementation device for a PCIe topology structure between Dies, It is characterized in that include: The routing table division unit is used to divide the original routing table into multiple routing sub-tables according to the number of dies in the PCIe switch and store them in the corresponding dies respectively, and each routing sub-table only includes the routing information of the port in the corresponding die; An inter-Die routing unit is used to match the egress port of the message using the routing sub-table of the first Die when receiving the message at the ingress port of the first Die. If the egress port is not matched, the message is forwarded to the second Die using the inter-Die transmission port, and the matching is continued using the routing sub-table of the second Die. If there is no match, the message is forwarded until the egress port is successfully matched and the message is sent out; The error processing unit is used to determine that a message error occurs when the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, return the message to the ingress port of the first Die, and report error processing information.

7. The routing implementation device for the inter-die PCIe topology structure according to claim 6, It is characterized in that Each Die includes the inter-Die transmission port, and the inter-Die transmission port includes an inter-Die input port and an inter-Die output port. The inter-Die routing unit is further used to: The message is sent to the inter-Die input port of the second Die through the inter-Die output port of the first Die.

8. The routing implementation device for the inter-die PCIe topology structure according to claim 7, It is characterized in that The routing subtable includes a plurality of port items and unmatched items, and the inter-die routing unit is further used to: If no output port is matched in the routing subtable of the second Die, the message is mapped to the unmatched item in the routing subtable through the inter-Die input port of the second Die, and then mapped to the inter-Die output port of the second Die through the unmatched item, and the message is continued to be forwarded through the inter-Die output port of the second Die.

9. The routing implementation device for the inter-die PCIe topology structure according to claim 6, It is characterized in that The number of routing failures is recorded in the descriptor of the message. When no egress port is matched in each routing sub-table, the number of routing failures is increased by 1.

10. The routing implementation device for the inter-die PCIe topology structure according to claim 9, It is characterized in that The error handling unit is further used for: When the number of routing failures is equal to the number of Dies in the PCIe switch, it is determined that the message has been forwarded to all Dies in the PCIe switch but still has not been matched to an egress port, that is, it is determined that a message error occurs.