PCIE (Peripheral Component Interconnect Express) retimer using inter-die data interface to implement redundant endpoint failover

Abstract

Receiving, by an upstream dummy port of a first circuit die of a multi-die integrated circuit module (ICM), a plurality of serial data channels associated with the PCIe data link; corresponding deserializing channel specific data code words are generated; providing the deserialized channel specific data codeword for transmission via a downstream dummy port of the first circuit die of the multi-die ICM, the downstream dummy port having a PCIe data link through to a first endpoint; in response to a failure of the PCIe data link through the first endpoint, rerouting the deserialized channel specific data codeword to a second circuit die of the multi-die ICM via an inter-die data interface employing an inter-die adaptation layer protocol; receiving, by the second circuit die, the deserialized channel specific data codeword from the inter-die data interface; and through a second downstream pseudo port, sending the deserialized channel specific data code word to a second endpoint through a second PCIe data link.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 382,900, filed November 9, 2022, and entitled “PCIe Retimer for Redundant Endpoint Failover Using Inter-Die Data Interface,” which is incorporated herein by reference in its entirety for all purposes. Background Art

[0003] With the data rate of PCIe 5.0 (32Gbps) increasing from previous generations (for example, PCIe 4.0 MAX has a data rate of 16Gbps), the channel reach is becoming shorter than before, and the need for retimers is becoming more apparent. Common channels include system boards, backplanes, cables, riser cards, and add-in cards. The loss of connections across these channels (usually a combination of such channels and their slots) often exceeds the target loss requirement of -36dB at 16GHz. Retimers can extend the channel reach beyond the reach limit when a retimer is not used.

[0004] The retimer splits the link between the host (root complex, abbreviated as RC) and the device (endpoint) into two independent segments. Therefore, the retimer can re-establish a new forward PCIe link, and this re-establishment process includes retraining and appropriate equalization for the physical layer and link layer.

[0005] Redrivers are purely analog amplifiers that boost signals to compensate for attenuation. However, while boosting the signal, redrivers also increase noise, which often exacerbates jitter. In contrast, retimers incorporate both analog and digital logic to equalize the signal, extract its clock timing information, and output a signal with high amplitude and low noise and jitter. Furthermore, retimers maintain power states to keep system power consumption low.

[0006] The retimer specification debuted in PCIe 4.0 and is expected to continue in PCIe 5.0. Figure 1 and Figure 2 Shown are common application scenarios of retimers in some embodiments. Figure 1 A retimer is used and placed on the motherboard, logically between the PCIe root complex (RC) and the PCIe endpoints.

[0007] Figure 2 The scenario shown uses two retimers, where the first retimer is also located on the motherboard, while the second retimer is located on a riser card that serves as a connection between the motherboard and an add-in card, where the PCIe endpoints are contained.

[0008] In complex PCIe systems, there may be far more PCIe endpoints than available PCIe ports. In such cases, a switch can be used to increase the number of PCIe ports. A switch connects multiple endpoints to a single root node and routes data packets to specific destinations, rather than mirroring data across all ports. A key feature of switches is bandwidth sharing, allowing all endpoints to share the root node's bandwidth. Summary of the Invention

[0009] The methods and systems described herein are configured to: receive, by a plurality of upstream serial data transceivers of a first circuit die of a multi-die integrated circuit module (ICM), a plurality of serial data lanes associated with a PCIe data link, and responsively generate corresponding deserialized channel-specific data codewords; provide the deserialized channel-specific data codewords for transmission via a set of downstream serial data transceivers of the first circuit die of the multi-die ICM, the set of downstream serial data transceivers having a PCIe data link to a first endpoint; reroute the deserialized channel-specific data codewords to a second circuit die of the multi-die ICM via an inter-die data interface employing an inter-die adaptation layer protocol in response to a failure of the PCIe data link to the first endpoint; receive, by the second circuit die, the deserialized channel-specific data codewords from the inter-die data interface; and transmit, by a second set of downstream serial data transceivers, the deserialized channel-specific data codewords via a second PCIe data link to a second endpoint.

[0010] This "Summary" section provides a brief summary of the concepts described in detail in the "Detailed Description" section below. This "Summary" section is not intended to provide key or primary features of the claimed technical solution, nor is it intended to assist in determining the scope of the claimed technical solution. Other objectives and / or advantages of the embodiments of the present invention will become readily apparent to those skilled in the art by referring to the "Detailed Description" section and the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 and Figure 2 Shown are two uses of a retimer in some embodiments.

[0012] Figure 3 FIG. 1 is a block diagram of a chip structure of a multi-die integrated chip module (ICM) that implements redundant endpoint failover through high-speed die-to-die (D2D) interconnection in some embodiments.

[0013] Figure 4 FIG. 1 is a data flow diagram for a multi-die ICM operating in retimer mode with data path routing within the same die in some embodiments.

[0014] Figure 5 FIG. 1 is a data flow diagram for a multi-die ICM operating in retimer mode with D2D interconnects for routing data paths between circuit dies in some embodiments.

[0015] Figure 6 FIG. 1 is a block diagram of a multiplexing crossbar switch used for data path routing in some implementations.

[0016] Figure 7 Schematic diagram of D2D interconnection in some implementations.

[0017] Figure 8 FIG. 4 is a block diagram of an adaptation layer for D2D interconnection in some implementation modes.

[0018] Figure 9 FIG. 4 is a block diagram of a chiplet-to-chiplet (T2T) serial peripheral interface (SPI) bus structure in a four-chiplet implementation.

[0019] Figure 10 FIG. 1 is a block diagram of the complete signal path between the central processing unit (CPU) core 900 and the PHY in each chip in a multi-chip module.

[0020] Figure 11 is a flow chart of a method in some embodiments. DETAILED DESCRIPTION

[0021] While the ability to fully integrate multiple systems into the same integrated circuit is increasing, maintaining multiple chip systems and subsystems separately still has significant advantages. For non-limiting descriptive purposes, at least some aspects of the present invention described herein are illustrated in a system environment comprising at least one point-to-point communication interface connecting two integrated circuit chips, representing a root complex (i.e., a host) and an endpoint, respectively, wherein the communication interface is supported by a number of data channels, each of which consists of four high-speed transmission line signal conductors.

[0022] A retimer typically consists of a physical physical layer (PHY) and retimer core logic. The PHY consists of a receiver and a transmitter. The PHY receiver is responsible for data reception, data deserialization, and clock recovery, while the PHY transmitter is responsible for data serialization and amplification before output transmission. The retimer core logic is responsible for de-skew (for multi-channel links) and rate adaptation to address frequency differences between ports on both sides.

[0023] Retimers offer additional value because they sit in the path between the root complex (e.g., CPU) and endpoints (e.g., cache blocks). Retimers can also incorporate integrated processing units (e.g., accelerators) to handle data processing in the path from the root complex to the endpoint.

[0024] To achieve a highly flexible solution, the PCIe retimer uses a conventional PHY interface for the PCIe bus and a high-speed die-to-die (D2D) interconnect for the data processing unit (DPU). The high-speed die-to-die interconnect structure enables extremely high-speed communication links between cores within the same package structure. The PCIe retimer circuit is a core (die) with a four-channel retimer and can be connected to the DPU core via a high-speed die-to-die interconnect. One, two, or four channels can form a multi-channel link, and data can be transmitted across all links. Alternatively, each channel can be constructed as a single-channel link. Each channel in the PCIe retimer has two PHYs, one at each end (i.e., the upstream port and the downstream port). In the case of four channels, a PCIe retimer die has eight PHYs. In addition, the PCIe retimer die also has communication lines to enable the exchange of control information between two or more PCIe retimer dies.

[0025] With one (or more) PCIe retimer cores, the following structures can be constructed (these structures are described in further detail below):

[0026] -Quad-channel retimer;

[0027] - Single die with full-flexibility 4×4 static channel routing;

[0028] - Four-channel retimer with accelerator (DPU);

[0029] Two dies in the same package: a retimer die and a DPU die.

[0030] - Eight-channel retimer;

[0031] Two dies within the same package structure, but with limited static channel routing – Flexible 4×4 routing within the same die, but without the ability to cross die boundaries;

[0032] - Eight-channel retimer with full-flexibility channel routing;

[0033] Two dies within the same package structure use high-speed inter-die interconnects to route data across different die, but this incurs additional latency.

[0034] - Eight-channel retimer with accelerator (DPU);

[0035] Three dies in the same package: two retimer dies and one DPU die.

[0036] - Sixteen-channel retimer;

[0037] Four dies within the same package structure, but with limited static channel routing – Flexible 4×4 routing within the same die, but without the ability to cross die boundaries.

[0038] Multi-die ICM for redundant endpoint failover via D2D

[0039] Redundancy is a characteristic feature of many electronic systems, typically used to ensure system reliability and continued operation in the event of a failure in a critical component or hardware device. Redundant systems and / or hardware devices can take over in the event of a failure until the primary system and / or hardware device can be repaired. In some environments, such as underwater data centers, frequent maintenance is unnecessary, requiring only scheduled, scheduled maintenance. In such environments, redundancy ensures normal operation between scheduled maintenance periods and reduces the need for emergency repairs.

[0040] Figure 3 The following is a block diagram of the chip architecture of a multi-die ICM 300 in some embodiments. As shown, multiple upstream serial data transceiver PHYs of an upstream pseudo port of a first circuit die 305 receive multiple serial data lanes associated with a PCIe data link. These multiple upstream serial data transceiver PHYs convert the received serial data lanes into corresponding deserialized channel-specific data codewords. Circuit die 305 is configured to route the deserialized channel-specific data codewords to a first endpoint 315 or a second endpoint 320. In at least one embodiment, first endpoint 315 is a primary endpoint, and second endpoint 320 is a redundant endpoint. The deserialized channel-specific data codewords are routed to the primary endpoint 315 via the PCIe data link by a group of downstream serial data transceiver PHYs of a downstream pseudo port within the first circuit die 305. If the PCIe data link to the primary endpoint 315 fails, the deserialized channel-specific data codewords are rerouted to the second circuit die 310 of the multi-die ICM 300 via the inter-die data interface using the inter-die adaptation layer protocol. After the second circuit die receives the deserialized channel-specific data codeword from the D2D interface, a group of downstream serial data transceivers PHY of the downstream pseudo port in the second circuit die sends the codeword to the second endpoint 320 via the second PCIe data link.

[0041] Figure 3The system includes a Board Management Controller (BMC) 325. The BMC may be included in a motherboard, for example, and utilizes sensors to monitor the status of components and hardware devices within the motherboard and transmit such device status to, for example, a root complex. The BMC may be used in server room / data center applications, and can be remotely managed by an administrator to access information related to the entire system. Some of the BMC's monitoring functions include temperature monitoring, humidity monitoring, power supply voltage monitoring, fan speed monitoring, communication parameter monitoring, and operating system monitoring. When any parameter exceeds a threshold, the BMC can notify the administrator, allowing the administrator to take action. In some embodiments, the BMC may be preconfigured to take certain actions when a parameter exceeds a threshold, such as (but not limited to) executing a sequence of operations to switch to a redundant endpoint in the event of a primary endpoint failure. In some embodiments, the BMC 325 monitors the status of the PCIe link between the first circuit die 305 and the primary endpoint 315. In such embodiments, monitoring the PCIe link status includes measuring the bit error rate of the upstream and downstream data paths. When the bit error rate exceeds a threshold, it may indicate a PCIe link failure in one or more lanes, potentially necessitating a link retraining sequence between circuit die 305 and endpoint 315. If multiple consecutive link retraining sequences fail, the PCIe link may be fatally faulted, and multi-die ICM 300 can establish a link between root complex 302 and redundant endpoint 320 using the D2D interface. In one such embodiment, routing logic in first and second circuit dies 305 and 310, respectively, can be configured to reroute traffic from the downstream serial data transceiver in first circuit die 305 via the D2D interface using a D2D adaptation layer protocol, as described in further detail below. Second circuit die 310 routes incoming traffic received by the D2D adaptation layer to the downstream serial data transceiver in the second circuit die for transmission to redundant endpoint 320. This allows normal operation to continue, for example, to allow the system to complete routine maintenance and restore functionality of the PCIe link to the primary endpoint. In some embodiments, the configuration state of the multi-die ICM 300 may indicate the point at which a failure occurred, and such configuration information may be accessed and read by, for example, the BMC 325, the root complex 302, or other diagnostic equipment to evaluate the system and / or hardware device to be repaired. In such embodiments, the configuration state may be obtained, for example, via an SMBus between devices, or may be communicated as a Vendor Defined Message (VDM) using a Control Skip Ordered Set.

[0042] The BMC may be used to provide instructions to the CPU of the master core of the ICM 300. Such instructions may be connected, for example, via an SMBus or various other point-to-point connections, and may be associated with a root complex to endpoint mapping relationship. The CPU of the master core may configure the channel routing logic of the master core and the slave core to map the upstream pseudo port to the downstream pseudo port associated with the mapping relationship instruction issued by the BMC. In some embodiments, the configuration of the channel routing logic includes modifying the configuration register space within the two circuit dies, wherein the configuration register space includes control signal values, such control signal values ​​being provided as selection signals to multiplexing devices within the channel routing logic. In some embodiments, as described below, the upstream pseudo port has a static mapping configuration content that is mapped to the adaptation layer port. For example, Figure 6 The upstream pseudo port PHY1 in the example can be statically mapped to adaptation layer port 1 of the master circuit die's adaptation layer transmit (TX) portion, while the downstream pseudo port of the slave circuit die can selectively connect to any of adaptation layer ports 0-7 based on the root complex-to-endpoint mapping. In some embodiments, the downstream pseudo port can be statically mapped to an adaptation layer port, while the upstream pseudo port can be configured to connect to any adaptation layer port, and vice versa.

[0043] In some embodiments, a failure in the PCIe link to the master endpoint 315 may be related to a connection (e.g., a trace on a PCB) between the multi-die ICM 300 and the master endpoint 315. In some embodiments, the master endpoint 315 may have a fault within any of its components and may need to be replaced during the next maintenance. In some embodiments, the master endpoint 315 may report to the BMC 325, for example, that a temperature fluctuation exceeds a threshold or that a bit error rate falls below a threshold, which may then initiate an operational sequence for activating a redundant endpoint. This operational sequence may be performed with or without administrator input and may include: transmitting a command over the SMBus to activate an active CPU core in the master circuit, reconfiguring the channel routing logic in the multi-die ICM, and writing to the configuration register space associated with the channel routing logic in the master and slave circuit dies, thereby routing data to the slave circuit dies via the D2D interface.

[0044] Figure 41 is a data flow diagram for a PCIe data link to a first endpoint 315 in some embodiments. As shown, a PHY within an upstream pseudo port receives serial data and includes a deserializer for converting the serial data stream into, for example, 32-bit channel-specific data codewords after deserialization. The data codewords are routed to the retimer core logic via channel routing logic. In some embodiments, the core logic includes a PCS decoding module for performing, for example, 8b / 10b or 128b / 130b decoding before storing in the retimer FIFO. The retimer FIFO includes channel de-skew and rate adaptation functionality across multiple channels within a given circuit die, as well as channel de-skew and rate adaptation functionality between channels across multiple different circuit dies. After reading the channel-specific data codewords from the retimer FIFO, the downstream serial data transceiver transmits them via a transmitter within the PHY of the downstream pseudo port.

[0045] Figure 5 3 is a data flow diagram for a PCIe data link to be used via a D2D interface. Specifically, Figure 5 As shown, data is first received by a set of serial data transceivers and then provided to the second circuit die via an inter-die data interface (such as an adaptation layer and an inter-die transmitter). Figure 5 In the example, data is received by the PHY of the first circuit die. After deserialization, the data is routed through the channel routing logic to the adaptation layer within the first circuit die, which formats the raw data for transmission over the D2D interface. The adaptation layer within the second circuit die receives the data and reverse-formats it to provide it to the target channel in the second circuit die. Before the data is output to the endpoint via the serial data transceiver PHY of the second circuit die, it is also provided to the RPCS logic for rate adaptation and inter-channel de-skew. A similar data path exists in the opposite direction from the endpoint to the root complex.

[0046] like Figure 4 and Figure 5 As shown, the RPCS logic may, for example, include 8b / 10b encoding / decoding functionality for PCIe Gen 1 and Gen 2, and 128b / 130b encoding / decoding functionality for PCIe Gen 3 through Gen 5. The embodiments described herein further contemplate PCIe Gen 6, which utilizes a flow control unit (FLIT) scheme, eliminating the need for 8b / 10b or 128b / 130b. In such embodiments, encoding / decoding functionality may be omitted, but at the same time, PCIe Gen 6-specific functionality, such as FEC decoding (both partial and full decoding), is further included in the data path in the form of logic. Some functionality of the retimer core logic is shared, such as FIFO inter-channel deskew and rate adaptation.

[0047] Figure 6 FIG. 6 is a block diagram of channel routing logic 600 of an ICM retimer circuit die in some embodiments. Figure 6 The diagram includes the left block diagram and the right diagram of various channel routing settings. As shown in the figure, the top channel routing setting 605 is a loopback mode, in which data is received by the PHY, deserialized and passed through the core routing logic, and then serialized and output by the same PHY back to the original source. The middle channel routing setting 610 corresponds to Figure 4 The data path shown in FIG. 6 is a data path where data always stays within the same die. As shown, data is received by the first PHY and provided to the second PHY after being processed by the core routing logic. The channel routing setup 615 corresponds to Figure 5 The data paths shown are where data is received by the PHY of the first circuit die, forwarded directly to the high-speed die-to-die interconnect, and then output via the PHY of the second circuit die. All three scenarios further have data paths in the opposite direction.

[0048] Figure 6 The multiplexing function of the channel routing logic is further shown. Any given PHY can be used to receive data from any of the eight channels. In addition, data can be obtained from the D2D data interface. Although Figure 6 The multiplexing circuitry for only one channel to the inter-die data interface is shown, however, it should be noted that equivalent multiplexing circuitry exists for all PHYs. For each adaptation layer port in the adaptation layer used for transmission over the high-speed die-to-die interconnect, any input PHY can be selected. In some embodiments, data mirroring can be implemented by selecting the same upstream PHY data for multiple adaptation layer physical ports, so that traffic received by an upstream PHY is replicated on multiple downstream PHYs at different pseudo-ports.

[0049] Data path switching in the routing logic involves: carrying the deserialized channel-specific data codewords on a 32-bit receive data bus; enabling the corresponding data lines; clock recovery; and the corresponding reset operation. It is important to note that only the raw data is multiplexed; no processing of the received data occurs. The raw data multiplexing logic is statically configured for data routing via configuration bits. Changing the raw data multiplexing settings during mission mode may result in invalid data and may cause glitches on the clock lines. Therefore, the multiplexing logic can be configured during reset.

[0050] In some embodiments, each circuit die includes channel routing logic, such as raw data multiplexing logic for performing channel routing between upstream pseudo ports and downstream pseudo ports of the same die or different circuit dies. In one such embodiment, the raw data multiplexing function within each circuit die can be configured by a master circuit die (also referred to as a "master device"), for example, by writing to a configuration register associated with the raw data multiplexing. Figure 9 and Figure 10 Such a chiplet-to-chiplet communication is shown. Figure 9 Figure 2 shows the T2T SPI bus architecture for a four-chiplet scenario. This specific number of chips is not limiting, and the principles described here can be extended to an N-chiplet retimer with one master chiplet and N-1 slave chipslets, where N ≥ 2.

[0051] The master T2T SPI 985 includes a serial clock line SCK, which carries the serial clock signal generated by the master T2T SPI 985. The SCK signal is received by all slave T2T SPIs and is used to coordinate the reading and writing of data over the T2T SPI bus.

[0052] The master T2T SPI 985 also includes a Leader Out Follower In (MOSI) line and a Leader In Follower Out (MISO) line. The MOSI line is used for sending data from the master device to the slave device (i.e., as part of a write operation), while the MISO line is used for sending data from the slave device to the master device (i.e., as part of a read operation).

[0053] The master T2T SPI 985 further includes a slave chip select Line, which is used to indicate which slave device will participate in the current operation of the bus, that is, which slave device the data or command in the bus is for. Figure 9 Only one wire is shown for selecting a line from the core particle, but in actual situations, there may be one wire for each line, that is, Figure 9 In the case shown, there may be three different slave die selection wires.

[0054] In addition, each slave T2T SPI 975a, 975b, 975c is connected to all the above lines to achieve bidirectional communication between the master T2T and the slave T2T, thereby achieving communication between the chiplets.

[0055] Figure 10 Shown is the complete signal path between the CPU core 900 and the PHY in each chip in a multi-chip module.

[0056] The CPU core 900 is connected to the PHY 970 within the master chiplet via the master chiplet APB interconnect 925, thereby enabling communication with the PHY 970 via the APB interconnect 925. The CPU core 900 is also connected to the master T2T SPI 985 via the master chiplet APB interconnect 925. The master T2T SPI 985 is part of the T2T SPI bus that enables the CPU core 900 to communicate with other chiplets.

[0057] like Figure 10 As shown, each slave chip comprises a corresponding slave T2T SPI 975a, 975b, 975c. Each slave SPI is connected to the master T2T SPI 985 to achieve signal transmission between the chiplets.

[0058] Each slave SPI 975a, 975b, 975c is connected to a corresponding PHY 970a, 970b, 970c via a corresponding slave chip APB interconnect 926, 927, 928. Each slave SPI 975a, 975b, 975c acts as a master device for the corresponding APB interconnect 926, 927, 928. In this way, each slave SPI can access all registers within the chip where the slave SPI resides.

[0059] As can be seen, communication between chiplets uses two different buses and protocols. The SPI protocol does not support addressing, but the APB protocol does. Part of the data that the CPU core 900 places on the T2T SPI bus is APB address information, which enables the local APB interconnect of each slave chiplet to route the message to the target receiving PHY.

[0060] Each PHY is assigned a unique APB address or APB address range, so that the CPU core 900 can read and write to a specific PHY in any chip. From the perspective of the CPU core 900, the entire multi-chip module has a single address space that includes different address regions for each PHY.

[0061] For ease of explanation, assuming that the APB address is 24 bits and the data codeword size is 32 bits, the control information provided to the SPI bus may take the following format, which is referred to herein as a "control packet".

[0062] r r r r r s s s a a a a a a a a a a a a a a a a a a a a a a a a

[0063] Among them, bits 0 to 23 are address bits ("a"), bits 24, 25, and 26 are slave core selection bits ("s"), and bits 27 to 31 are reserved bits ("r"). In this specific case, since there are three slave cores (that is, the number of slave T2T SPIs is three), the number of slave core selection bits is three. The reserved bits leave room for other slave core selection bits - in this case, up to eight slave core selection bits can be provided, thus supporting a maximum of eight slave cores. This principle can be further extended to any number of slave cores by increasing the codeword size.

[0064] The address bits constitute the APB address. Each slave T2T-SPI is configured as the master bus for its corresponding local APB interconnect, enabling each slave T2T-SPI to instruct its corresponding APB interconnect to perform write or read operations on the corresponding PHY connected to each APB bus. In some cases, address data may not be used, and instead a T2T-SPI bus may be employed that automatically increments the address to obtain the address to which data is to be written. The address data may be provided to the local APB interconnect after the corresponding slave T2T SPI receives the control packet, enabling the local APB interconnect to route commands and data to the correct local PHY.

[0065] The slave chip selection bit enables the control packet to indicate the slave chip selection line to be started, that is, the chip to be read or written. The T2T SPI bus controls the slave chip selection line through the slave chip selection bit. For example, 0 indicates that the corresponding slave chip selection circuit should be at a low potential, and 1 indicates that the corresponding slave chip selection circuit should be at a high potential.

[0066] Alternatively, the slave chip select control information can be sent separately from the APB address data. This information can be sent in-band as described above, or using another channel such as the System Management Bus (SMBus). The address data can be sent separately before the data packet is sent. In some cases, address data can be omitted and a T2T-SPI bus can be used, which automatically increments the address to determine the address of the data to be written.

[0067] In each of these scenarios, data can be sent after the slave chip select and address information are provided (the address information may or may not be provided, depending on the specific needs). The master T2T SPI 985 can maintain the slave chip select line set until a new instruction is received regarding the configuration of the slave chip select line. Similarly, the associated APB interconnect can continue to write to the specified address (or auto-increment address) until new addressing information is provided. In this way, data and commands can be sent or received to any PHY in any chip.

[0068] The APB address space is a global address space across all corelets. That is, any register in any corelet can be addressed through this global address space. In a specific structure, each corelet is provided with a base address in the form of a corelet identifier multiplied by a constant. The corelet identifier can be the corelet number, and the constant can be the base address of the main corelet. In addition to this, other storage space structures can also be used. Each register in each corelet is assigned a unique address or address range within the global address space. In this way, each PHY in PHY 970, 970a, 970b, 970c is assigned a unique address or address range.

[0069] The CPU core in the master core can coordinate the channel switching circuits in the two cores. The CPU core in the slave core can be in a low-power state. As shown in the figure, the SPI communication bus between the two cores can be used to configure the switching circuit in the slave core to select between the first and second groups of downstream serial data transceiver ports. In some embodiments, a die-to-die (D2D) interface can be set up and the channel routing between the master core and the slave core can be configured using the D2D interface, that is, the serial data stream received by the upstream port of the master core is routed to the downstream port of the slave core, and vice versa. Such a D2D interface can also be used to transmit configuration information as sideband information from the master core to the slave core, for example, to configure the configuration registers of the slave core. In another embodiment, the raw data cross-multiplexer can be configured via a system management bus, which can also be connected to the root complex. In some embodiments, the configuration purpose can be achieved through a virtual channel between the root complex and the retimer chip. In such an embodiment, a vendor-defined message (VDM) may be included in a specific vendor-defined packet field of a PCIe data transmission. Such a VDM may be detected and extracted, for example, via an interrupt protocol and provided to the CPU of the master die. Figure 7 There is only a single slave chiplet, but it should be noted that other slave chips may be included, and in some embodiments, up to three slave chips may be included. In such a case, each slave chiplet may have a specific chiplet ID, and configuration register write commands may be assigned to specific chiplet IDs.

[0070] In some embodiments, the master chiplet can initialize the configuration registers of the slave chiplet's raw data multiplexer so that the receive (RX) adaptation layer port is statically mapped to the downstream port leading to the redundant endpoint. In one such embodiment, the master chiplet can switch the routing of the deserialized channel-specific data codeword between: (i) the downstream port leading to the master endpoint within the same die and (2) the adaptation layer to be routed via the D2D interface.

[0071] In some embodiments, when a PCIe link to a primary endpoint fails, the deserialized channel-specific data codewords are rerouted via the D2D interface. Such link failures may be related to a failure of the primary endpoint itself, so the configuration register settings of the circuit die within the ICM may be useful for diagnostic purposes. For example, the root complex and / or the ICM may obtain and provide configuration parameters indicating that the PCIe data link to the backup endpoint is up, indicating that the primary endpoint itself or a portion of the PCIe data link to the primary endpoint may require repair. In such a scenario, the primary endpoint can be repaired or replaced, and the ICM can be used to restart the PCIe data link to the primary endpoint.

[0072] Figure 7 1 is a block diagram of an inter-die data interface (also referred to herein as a "high-speed die-to-die (D2D) interconnect," a "D2D link," etc.) in some embodiments. As shown, the D2D link has eight high-speed die-to-die (D2D) data streams, four in each direction. Each D2D data stream operates at a rate of 25 GBd, sending 5 bits over 6 wires, for a total throughput of 125 Gbps. In some embodiments, each D2D data stream uses an orthogonal differential vector signaling (ODVS) encoding scheme to encode 5 bits into 6 bits for transmission over 6 wires. In addition, the interface includes two differential clock channels operating at a frequency of 6.25 GHz. Note that in addition to these, interconnects with other sizes, APB interconnect structures, and / or encoding schemes may also be used.

[0073] As shown in the figure, the original bandwidth of each D2D data stream is within 125Gbps without forward error correction (FEC), and with FEC, the bandwidth is 125Gbps×150 / 160=117.1875Gbps. In some embodiments, the PCIe retimer uses low-latency FEC and scrambling schemes for high-speed die-to-die interconnect operations. In this structure, 150 bits of data are sent in each clock cycle of each data stream. The clock frequency may depend on the link speed. At 125Gbps, the core clock frequency is 125Gbps / (5×32)=781.25MHz. Among them, the 150 bits of data sent at one end of the link are aligned at the receiving end, that is, TX bit 0 is received as RX bit 0. The 150 bits of data in one clock cycle are called a "codeword".

[0074] The inter-die data interface operates using the same 100 MHz reference clock as the PHY. In some embodiments, this interface is configured using an APB interface with an 8-bit wide data bus. In some embodiments, this interface can be used to reduce power consumption by reducing speed. Furthermore, the number of enabled TX / RX data streams can be adjusted based on the required bandwidth for communication.

[0075] Figure 8 FIG. 1 is a block diagram of an adaptation layer (AL) for an inter-die data interface in some embodiments. The adaptation layer formats the payload sent and received via the high-speed die-to-die interconnect. Figure 8 As shown, the Adaptation Layer supports the following types of payloads:

[0076] Raw SERDES RX data (up to eight SERDES);

[0077] Data frames / packets from a link controller that supports flow control (up to eight interfaces in operation);

[0078] Indirect register read and write commands executed through the APB bus.

[0079] exist Figure 3 In one embodiment, the retimers 305 and 310 may be used Figure 5 The retimer data path is shown in Figure 5 In one such embodiment, to minimize latency, the original data is sent from the original interface over the D2D interface.

[0080] Original data format

[0081] In some embodiments, each retimer circuit die includes eight PHYs. In some embodiments, all eight PHYs are connected to the root complex, and eight lanes of traffic are sent over the D2D interface. In such embodiments, the eight raw SERDES RX data interfaces operate in parallel. Depending on the protocol, the eight data frame interfaces can operate in round-robin fashion or in parallel. The high-speed link is statically configured to transmit either raw SERDES RX data or data frames. Indirect register accesses can be interleaved within both traffic types.

[0082] like Figure 8 As shown in FIG, the original SERDES RX data stream receives two 32-bit data codewords from the SERDES in two consecutive receive clock cycles and writes the combined 64-bit data into the asynchronous FIFO. The 64-bit data from the asynchronous FIFO is transmitted through a specific data flow of a high-speed die-to-die link (such as Figure 7If all eight channels are sent via the D2D interface, the raw data received from the two RX SERDES asynchronous FIFOs is combined and sent via the same specific data stream on the high-speed link. If only four data channels are sent via the D2D interface, there are two scenarios. In the first scenario, the data from the two channels is combined into a single D2D data stream in a similar manner, resulting in two D2D data streams that are not used for data transmission (but can be used, for example, for T2T configuration information). In the second scenario, the data from each channel is sent via its own D2D data stream.

[0083] The original data format (i.e., non-frame-based protocol) is a format used to transmit the original 32-bit SERDES data group in each data stream clock cycle. The codewords of the non-frame-based protocol are as follows:

[0084] 149 148:0 protocol Payload

[0085] The protocol bit of the non-frame protocol is set to 1'b1. As shown in Table 1 below, the format of bits 148:0 of ​​the payload field is as follows:

[0086]

[0087]

[0088] Table 1

[0089] The SERDES payload has a high priority, the register command has a medium priority, and the priority for future messages has a low priority. The SERDES payload can be inserted into the user data cycle, starting with PAYLOAD0, followed by PAYLOAD1, and so on. Register commands can only be inserted when there are fewer than four SERDES payloads already prepared in the data stream cycle. Register commands are inserted only into the PAYLOAD3 field. Before a new register write address command is issued, the register write address command precedes the register write data command. A register read address command or a register read data command can be inserted between a register write address command and a register write data command.

[0090] Although the above detailed description Figure 7 and Figure 8The specific D2D interface shown is shown, but it should be noted that other D2D interfaces can also be used to transmit PCIe traffic between circuit dies. In some embodiments, the D2D interface can be a Universal Chiplet Interconnect Express (UCIe) interface. UCIe has multiple operating modes, one of which is a FLIT-aware operating mode that executes protocols such as CXL / PCIe with a die-to-die adapter. In addition, UCIe also has a streaming protocol that provides a common mode for sending raw data for user-defined protocols. In combination Figure 3 In the multi-endpoint switching implementation described above, such a streaming protocol may be used to transfer data between circuit dies in a retimer operating mode.

[0091] Load distribution: non-load balancing mode

[0092] The protocol allows for payloads to be sent over the D2D link in either load-balanced or non-load-balanced mode. All data flows use one or the other operating mode. When the D2D link is sending PCS payload data (raw SERDES data), it uses non-load-balanced mode.

[0093] In raw SERDES mode, payload data from a fixed set of channels is statically configured for transmission via a specific D2D data flow. In this case, a "logical channel" corresponds to an adaptation layer physical port, which is the PHY mapping target of the raw data multiplexing crossbar switch. To minimize multiplexing logic, a fixed logical channel-to-data flow mapping is used. In one embodiment, traffic from eight channels of the adaptation layer physical port to four die-to-die data flows is mapped as follows:

[0094] Logical channels 0 to 1 are mapped to data stream 0

[0095] Logical channels 2 to 3 are mapped to data stream 1

[0096] Logical channels 4 to 5 are mapped to data stream 2

[0097] Logical channels 6 to 7 are mapped to data stream 3

[0098] This mapping also applies to non-SERDES payload data. Register commands and message payloads are statically configured to use a specific data flow, minimizing the logic required by processing only one command per cycle. Message payloads can be configured to use a different data flow than register commands.

[0099] In custom frame-based mode, similar to the original SERDES mode, each channel can be statically configured to a specific D2D data stream in the same manner as described above. For each D2D data stream's two data frame interfaces, the load from these two interfaces is distributed among the D2D link codewords in a round-robin manner. In some embodiments, for the same data frame interface / port in the same data stream, a minimum interval can be used between D2D link codewords. In some embodiments, this minimum interval can be four cycles.

[0100] In some embodiments, fixed time division multiplexing (TDM) time slots can be programmed. In fixed TDM mode, the transmitter sends codewords in a constant manner to the four ports it supports, such as port 0, port 1, port 2, port 3, port 0, port 1, etc. If a port has no payload to send in a certain time slot, it sends an "idle" cycle. In some embodiments, the number of ports in the TDM timing mechanism can be adjusted programmatically. In addition, similar to the original SERDES mode, register commands and message payloads can be statically set to use specific data streams to ensure that only one command needs to be processed in a cycle, thereby minimizing the required logic. Message payloads can be configured to use a different data stream than register commands.

[0101] APB master-slave interface

[0102] like Figure 8 As shown in the adaptation layer block diagram in , the D2D interface includes an APB slave interface and an APB master interface. The APB slave interface is an interface to all configuration registers of the adaptation layer, including configuration registers for setting core-to-core (T2T) read and write transactions. T2T transactions refer to indirect register read and write commands sent over the D2D link. The source of the T2T transaction is the adaptation layer in the master core, and the destination is the adaptation layer in the slave core for converting the received T2T read and write commands into APB read and write transactions. Both the APB slave interface and the master interface have a command FIFO, but only the APB master interface has a read return FIFO. The number of entries in these two types of FIFOs can be independent of each other, but in at least one embodiment, these two types of FIFOs are set to have the same size.

[0103] The APB master interface executes the received T2T read and write commands in the APB within the slave chip. For read commands, the corresponding read return data is sent back to the master chip via the D2D link. The command FIFO in the APB master interface allows for some "significant" write operations that may take time for the slave chip to complete. The firmware ensures that the command FIFO does not time out. The FIFO fill level can be read from a register, and the firmware ensures that timeouts do not occur by adding a delay between T2T write transactions or by performing a read after issuing a maximum number of consecutive T2T write transactions and then waiting for read data. The maximum number is determined by the number of command FIFO entries minus one. Because commands do not overwrite each other, T2T read transactions are used to flush the command FIFO. The master chip's APB master interface is idle, meaning it does not receive T2T transactions from the slave chip. The slave chip's APB slave interface is used to access the adaptation registers, but the slave chip does not initiate any T2T transactions.

[0104] T2T transactions

[0105] In some embodiments, the D2D interface has sufficient bandwidth to support in-band data transfers performed by the master chiplet in the MCM, for example, to configure other retimer chiplets or DPUs / accelerators connected via the D2D interface. T2T transactions are codeword addresses, that is, address bits 1:0 are zero. T2T transactions are read and written in a deterministic order. Any read or write operation to a slave chiplet register will not override other read or write operations to the same slave chiplet register. The master chiplet controls T2T transactions through five configuration registers within it, as shown in Table 2 below:

[0106]

[0107]

[0108] Table 2

[0109] In some implementations, the AL_T2T_CENTRY field and the AL_T2T_RENTRY field are located at the same register address to speed up access by enabling both fields to be read in the same operation.

[0110] The CPU of the master chiplet writes registers to the slave chiplet by executing the following APB transaction on the configuration registers in the adaptation layer over the D2D link:

[0111] 1) Write the APB register address value of the slave chip into AL_T2T_WADDR;

[0112] 2) Write the corresponding write data value from the chip into AL_T2T_WDATA to start executing the write command. As AL_T2T_WDATA is written, the AL_T2T_WADDR register automatically increments.

[0113] 3) If you need to continuously write the APB address of the slave chip, repeat step 2).

[0114] As long as the firmware can ensure that the command FIFO does not time out in the manner described above, step 3) can be repeated indefinitely. In some embodiments, the automatic increment of AL_T2T_WADDR can be disabled via a configuration bit. In one such embodiment, the configuration bit can be located in the same location as the AL_T2T_WADDR field as a new field.

[0115] The CPU of the master chiplet performs a register read operation on the slave chiplet by executing the following APB transaction on the configuration register in the adaptation layer via the D2D link:

[0116] 1) Read the AL_T2T_RENTRY register (unless the value is already known from a previous read operation);

[0117] 2) Write the APB register address value from the chip to AL_T2T_RADDR to start executing the read command. AL_T2T_RENTRY allows up to seven "major" read commands to be started before the next step starts.

[0118] 3) Waiting for read return data by polling the AL_T2T_RENTRY register. Once a non-zero value is read, it indicates the number of valid 32-bit data codewords prepared in the read return FIFO.

[0119] 4) Read the AL_T2T_RDATA register the number of times indicated by AL_T2T_RETNERY to obtain the data read from the chip.

[0120] Figure 111 is a flowchart of a method 1100 in some embodiments. As shown, method 1100 includes receiving 1105, by a plurality of upstream serial data transceivers of a first circuit die of a multi-die integrated circuit module (ICM), a plurality of serial data lanes associated with a PCIe data link, and generating corresponding deserialized channel-specific data codewords. The method further includes providing 1110 the deserialized channel-specific data codewords for transmission via a set of downstream serial data transceivers of the first circuit die of the multi-die ICM, the set of downstream serial data transceivers having a PCIe data link to a first endpoint. The method further includes rerouting 1115 the deserialized channel-specific data codewords to a second circuit die of the multi-die ICM via an inter-die data interface employing an inter-die adaptation layer protocol when the PCIe data link to the first endpoint fails. The method further includes recovering 1120, by the second circuit die, the deserialized channel-specific data codewords from the inter-die data interface. The method further includes sending 1225 the deserialized channel-specific data codewords to the second endpoint via the second PCIe data link by the second set of downstream serial data transceivers.

[0121] In some embodiments, method 1100 further includes detecting a failure of the PCIe data link at least in part via a BMC. In such embodiments, the deserialized channel-specific data codeword is rerouted in response to a command received from the BMC. In some embodiments, the command is received via a system management bus (SMBus). The BMC may monitor the status of a channel between the set of downstream serial data transceivers and the first endpoint. In some embodiments, the BMC monitors the performance of the first endpoint. Some performance characteristics representative of the performance monitored by the BMC may include, but are not limited to, parameters such as bit error rate, temperature, humidity, fan speed, and power supply voltage.

[0122] In some embodiments, the failure of the PCIe data link is associated with a channel interruption associated with the set of downstream serial data transceivers having a PCIe data link to the first endpoint. Such a channel interruption may, for example, be a failure in a trace, wire, or cable interconnecting the retimer and the endpoint. In some embodiments, detection of such a channel interruption may, for example, include: initiating a timeout by a retimer training and status state machine (RTSSM). The timeout may be initiated when the downstream pseudo port of the retimer does not receive input data within a predetermined period of time. Alternatively, the timeout may be initiated when an instruction from the first endpoint indicates that the first endpoint has not received output data within a predetermined period of time. The instruction from the first endpoint may be received by the retimer and / or the BMC via the system management bus (SMBus).

Claims

1. A method, characterized in that include: receiving, by a plurality of upstream serial data transceivers of a first circuit die of the multi-die integrated circuit module, a plurality of serial data lanes associated with a PCIe data link and responsively generating corresponding deserialized lane-specific data codewords; providing the deserialized lane-specific data codeword for transmission via a set of downstream serial data transceivers of the first circuit die of the multi-die integrated circuit module, wherein the set of downstream serial data transceivers has a PCIe data link to a first endpoint; In response to a failure of the PCIe data link to the first endpoint, rerouting the deserialized channel-specific data codeword to a second circuit die of the multi-die integrated circuit module via an inter-die data interface, wherein the inter-die data interface utilizes an inter-die adaptation layer protocol; receiving, by the second circuit die, the deserialized channel-specific data codeword from the inter-die data interface; and The deserialized channel-specific data codeword is sent to the second endpoint via the second PCIe data link through the second group of downstream serial data transceivers.

2. The method according to claim 1, wherein Also includes: The failure of the PCIe data link is detected at least in part by a card management controller, wherein the deserialized lane-specific data codewords are rerouted in response to receiving an instruction from the card management controller.

3. The method according to claim 2, wherein The command is received via the system management bus.

4. The method according to claim 2, wherein Also includes: The board management controller monitors a channel status between the group of downstream serial data transceivers and the first endpoint.

5. The method according to claim 2, wherein Also includes: The performance of the first endpoint is monitored by the board management controller.

6. The method according to claim 1, wherein The failure of the PCIe data link is associated with a lane outage associated with the set of downstream serial data transceivers having the PCIe data link to the first endpoint.

7. The method according to claim 4, wherein Also includes: An interruption of the channel is detected.

8. The method according to claim 5, wherein Detecting the channel interruption includes initiating a timeout by a retimer training and status state machine.

9. The method according to claim 6, wherein The timeout is initiated in response to no further input data being received within a predetermined period of time.

10. The method according to claim 6, wherein The timeout is initiated by an instruction from the first endpoint, wherein the instruction indicates that the first endpoint has not received any more output data within a predetermined period of time.

11. The method according to claim 8, wherein The command from the first endpoint is received via a system management bus.

12. The method according to claim 1, wherein The failure of the PCIe data link is associated with a monitored bit error rate exceeding a threshold.

13. A device, characterized in that: include: a plurality of upstream serial data transceivers of a first circuit die of the multi-die integrated circuit module, wherein the plurality of upstream serial data transceivers are configured to receive a plurality of serial data lanes associated with a PCIe data link and responsively generate corresponding deserialized lane-specific data codewords; a set of downstream serial data transceivers on the first circuit die for transmitting the deserialized channel-specific data codewords, wherein the set of downstream serial data transceivers has a PCIe data link to the first endpoint; channel routing logic on the first circuit die and the second circuit die, wherein the channel routing logic is configured to reroute the deserialized channel-specific data codeword to the second circuit die of the multi-die integrated circuit module via an inter-die data interface in response to a failure of the PCIe data link to the first endpoint, wherein the inter-die data interface utilizes an inter-die adaptation layer protocol; The second circuit die is used to restore the deserialized channel-specific data codeword from the inter-die data interface, wherein the second circuit die includes a second set of downstream serial data transceivers for sending the deserialized channel-specific data codeword to a second endpoint via a second PCIe data link.

14. The device according to claim 11, wherein It also includes a board management controller, which is used to detect the failure of the PCIe data link.

15. The device according to claim 12, wherein The board management controller is configured to provide instructions to a central processing unit of one of the first circuit die and the second circuit die of the multi-die integrated circuit module to configure the channel routing logic to reroute the deserialized channel-specific data codewords.

16. The device according to claim 12, wherein The board management controller is further configured to monitor a channel status between the group of downstream serial data transceivers and the first endpoint.

17. The device according to claim 12, wherein The board management controller is further configured to monitor the performance of the first endpoint.

18. The device according to claim 11, wherein One of the first circuit die and the second circuit die of the multi-die integrated circuit module includes a central processing unit for receiving instructions via a system management bus, wherein the central processing unit configures the channel routing logic on the first circuit die and the second circuit die to reroute the deserialized channel-specific data codewords.

19. The device according to claim 11, wherein The failure of the PCIe data link is associated with a lane outage associated with the set of downstream serial data transceivers having the PCIe data link to the first endpoint.

20. The device according to claim 17, wherein A retimer training and status state machine is also included for detecting the channel interruption and initiating a timeout.

21. The device according to claim 18, wherein The timeout is initiated in response to the first circuit die not receiving any further input data within a predetermined period of time.

22. The device according to claim 11, wherein The first endpoint is configured to issue a timeout to indicate that the first endpoint has not received any more output data within a predetermined time period.

Images