On-die chip-to-chip (C2C) link state monitor

Abstract

Techniques for chip-to-chip (C2C) serial communication, such as communication between chiplets on a multi-chip package, are disclosed. In some aspects, a method of monitoring on a die of a C2C link includes detecting a change of the C2C link from a first link state to a second link state, and storing link state change information in a first-in-first-out (FIFO) buffer on the die. The link state change information indicates the first link state, a duration of time that the C2C link was in the first link state, and a speed of the C2C link in the first link state. Upon detecting a request for the link state change information, the link state change information is retrieved from the FIFO buffer and sent in a serial fashion to an output pin of the die, such as a general purpose input / output (GPIO) pin.

Description

BACKGROUND 1. TECHNICAL FIELD

[0002] Aspects of the present disclosure relate generally to computer chip design, and in particular to chip-to-chip (C2C) communications, such as communications between chiplets on a multi-chip package.

[0003] 2. Related Art Description

[0004] Chiplets are a part of a processing module that make up a larger integrated circuit, such as a computer processor. Chiplets allow manufacturers to use multiple smaller chips to make up a larger integrated circuit, rather than manufacturing a processor on a single silicon die with the desired number of cores. Multiple chiplets working together in a single integrated circuit are referred to as a multi-chip module (MCM). Chiplets are becoming increasingly popular in different market segments due to advantages of chiplets over larger monolithic chips, such as improved performance and higher wafer processing yields.

[0005] Chip-to-chip (C2C) communications, such as communications between chiplets on a multi-chip package, typically occur on conductive traces on or within a package substrate to which the chiplets or dies are attached. Die-to-die (D2D) communications are similar, and the term C2C will be used hereinafter to refer to either or both C2C and D2D. C2C link architectures are influenced by the Peripheral Component Interconnect Express (PCIE) physical link layer and data link layer.

[0006] However, C2C or D2D links can not have external visibility for electrical analysis or protocol analysis, meaning that a protocol analyzer cannot be used because the link traces are not visible on the chip package. SUMMARY

[0007] The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary is merely presented in a simplified form to present some concepts relating to one or more aspects disclosed herein before the detailed description is presented. Thus, the following summary is not intended to be implemented or realized prior to the detailed description is presented.

[0008] In one aspect, a method of on-die monitoring of chip-to-chip (C2C) serial link status includes detecting a change in link status of the C2C serial link from a first link status to a second link status, storing link status change information in an on-die first-in-first-out (FIFO) buffer, the link status change information including information indicative of the first link status, a duration of the C2C serial link being in the first link status, and a speed of the C2C serial link in the first link status, detecting a request for link status change information, retrieving the link status change information from the FIFO buffer, and sending the link status change information in a serial fashion to an output pin.

[0009] In one aspect, an apparatus for on-die monitoring of C2C serial link training status and link status includes a die including a C2C serial link circuit including a physical layer configured to transmit and receive data via a C2C serial link according to a C2C communication protocol, a serial link state machine (SLSM) configured to perform link training of the C2C serial link, and a SLSM monitor coupled to the SLSM and configured to detect a change in link status of the C2C serial link from a first link status to a second link status, store link status change information in an on-die FIFO buffer, the link status change information including information indicative of the first link status, a duration of the C2C serial link being in the first link status, and a speed of the C2C serial link in the first link status, detect a request for link status change information, retrieve the link status change information from the FIFO buffer, and send the link status change information in a serial fashion to an output pin of the die.

[0010] In one aspect, an apparatus includes means for detecting a change in link status of a C2C serial link from a first link status to a second link status, means for storing link status change information in an on-die FIFO buffer, the link status change information including information indicative of the first link status, a duration of the C2C serial link being in the first link status, and a speed of the C2C serial link in the first link status, means for detecting a request for link status change information, means for retrieving the link status change information from the FIFO buffer, and means for sending the link status change information in a serial fashion to an output pin.

[0011] In one aspect, a non-transitory computer-readable medium storing computer- executable instructions that, when executed by an apparatus, cause the apparatus to: detect a change in a link state of a C2C serial link from a first link state to a second link state; store link state change information in an on-die FIFO buffer, the link state change information including information indicating the first link state, a duration that the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state; detect a request for the link state change information; retrieve the link state change information from the FIFO buffer; and send the link state change information to an output pin in a serial fashion.

[0012] In one aspect, information of a C2C link state, including the actual state of the state machine, the validity of the state, and the speed of the link, is output on a general purpose input / output (GPIO) pin that can be monitored using a logic analyzer. In this way, the C2C link state of both link partners can be monitored simultaneously.

[0013] Other objects and advantages associated with aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings are presented to aid in the description of various aspects of the disclosure and are provided solely for illustration of the aspects and are not intended to limit the aspects in any way.

[0015] Figure 1 Top level states of a serial link state machine (SLSM) are illustrated, and specifically, a PCIE link training status and state machine (LTSSM) is illustrated.

[0016] Figure 2 Chip-to-chip (C2C) communication using an on-die C2C SLSM monitor (SLSMM) in accordance with aspects of the disclosure is illustrated.

[0017] Figure 3 An example C2C SLSMM in accordance with aspects of the disclosure is illustrated in more detail.

[0018] Figure 4A 、 Figure 4B and Figure 4C are graphs each illustrating example C2C SLSMM serial output in accordance with aspects of the disclosure.

[0019] Figure 5 is a flow diagram illustrating a portion of an example process associated with an on-die C2C SLSMM in accordance with aspects of the disclosure. DETAILED DESCRIPTION

[0020] Techniques for chip-to-chip (C2C) serial communication, such as communication between chiplets on a multi-chip package, are disclosed. In some aspects, a method of monitoring on-die for a C2C link includes detecting, at an on-die C2C link circuit of a die for communicating via a C2C link according to a C2C protocol, a change of the C2C link from a first link state to a second link state, and storing link state change information in a first-in-first-out (FIFO) buffer on the die. The link state change information indicates the first link state, a duration of time that the C2C link was in the first link state, and a speed of the C2C link in the first link state. Upon detecting a request for the link state change information, the link state change information is retrieved from the FIFO buffer and sent in a serial fashion to an output pin of the die, such as a general purpose input / output (GPIO) pin.

[0021] Aspects of the disclosure are provided in the following description and related drawings, which are provided for exemplification purposes only. Alternative aspects can be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described or will be omitted so as not to obscure the relevant details of the disclosure.

[0022] The words “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage or mode of operation.

[0023] Those of skill in the art would understand that information and signals described herein can be represented using any of a variety of different technologies and / or methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof, depending in part on the particular applications, the exact design, and / or the corresponding technologies or modalities of the various aspects of the disclosure.

[0024] Further, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions being executed by one or more processors, or by a combination of both. Additionally, the sequence(s) of actions described herein can be considered to be embodied entirely within any form of non- transitory computer readable storage medium having stored therein a corresponding set of computer instructions that, upon execution, would cause or instruct an associated processor of a device to perform the functionality described herein. Thus, the various aspects of the disclosure can be embodied in a number of different forms, all of which have been contemplated to be within the scope of the claimed subject matter. In addition, for each of the aspects described herein, the corresponding form of any such aspects can be described herein in terms such as "logic configured to" perform the actions described, or other similar terminology. For the avoidance of doubt, the actions described herein are intended to be fully encompassed by the terms "configured to" and / or "logic configured to” unless otherwise indicated.

[0025] A die is a part of a processing module that makes up a larger integrated circuit, such as a computer processor. Dic allow manufacturers to use multiple smaller chips to make up a larger integrated circuit, rather than manufacturing a processor on a single silicon die with the desired number of cores. Multiple dice working together in a single integrated circuit are referred to as a multi-chip module (MCM). Due to the advantages of dice over larger monolithic chips, such as improved performance and higher wafer processing yields, dice are becoming increasingly popular in different market segments.

[0026] Chip-to-chip (C2C) communication, such as communication between dice on a multi-chip package, typically occurs on conductive traces on or within a package substrate to which the dice or dies are attached. Die-to-die (D2D) communication is similar, and the term C2C will be used hereinafter to refer to either or both of C2C and D2D. In some aspects, the C2C link architecture is implemented with influence of the Peripheral Component Interconnect Express (PCIE) physical link layer and data link layer.

[0027] Figure 1 A top-level state of a serial link state machine (SLSM) is illustrated, and specifically a PCIE link training state machine (LTSSM) 100 is illustrated. The LTSSM states are defined by the PCIE specification. The LTSSM state machine and the PCIE protocol are used herein as an illustrative example of how serial communication between link partners is implemented, and are not limiting: the same principles described herein can be applied to other protocols and other state machines.

[0028] Referring now to the PCIE LTSSM as an illustrative example, the top level states include Detect, Polling, Configuration, L0, L0s, LI, L2, Recovery, Loopback, Hot Reset, and Disabled. Each state is composed of sub-states, which together make up the state. The first LTSSM state after reset is the Detect state. Detect, Polling, and Configuration are link training states; Recovery is a link retraining state; L0, LI, and L2 are power management states; L0s is an active power management state, as is LI. Loopback, Hot Reset, and Disabled are other states.

[0029] Table 1 lists the LTSSM sub-states defined in the PCIE specification. The numerical assignment of sub-states is implementation dependent and is included as an illustration of a possible implementation. The numbers are six hexadecimal digits.

[0030] Table 1 - LTSSM Substates for PCIE

[0031]

[0032]

[0033] The PCIE states listed in Table 1 illustrate the point that a high-speed serial protocol can have the following characteristics: the protocol starts initially at a low rate over a serial link, and then uses a state-based handshake to reach the highest data rate achievable under current conditions, where the sequence of particular states and state transitions is defined by the protocol. In some cases, the protocol can define a minimum time that a device must remain in a particular state before being allowed to enter the next state. In some cases, the protocol can define a maximum time that a device can remain in a particular state before entering an error recovery state.

[0034] High-speed, multi-gigabit interfaces are extremely complex and challenging to product functionality and reliability. Many registers must be tuned for optimal performance, and the serial link state machine must be probed for link startup, link debug, diagnostics, and link training optimization. In addition, serial link state monitoring will be needed to isolate power, system, thermal noise, or design sensitivity issues for customer site fault diagnosis. The link speed needs to be monitored, as well as the underlying serial link state machine state and sub-state transitions and the time spent in each state, which directly impacts the total initialization time and thus system throughput.

[0035] However, C2C or D2D links can not have external visibility for electrical analysis or protocol analysis, meaning that a protocol analyzer cannot be used because the link traces are not visible on the chip package. If implemented as an alternative, complex and expensive on-chip debug logic setup will be required.

[0036] Figure 2 C2C implementations with on-die SLSM monitors and block diagrams of SLSM monitor logic are illustrated in accordance with aspects of the present disclosure. In Figure 2 In the example, on a substrate 200, two die, die 1 202 and die 2 204, are mounted. Die 1 202 includes a first C2C serial transceiver C2C1 206 coupled to a first physical layer circuit PHY1 208. C2C1 206 includes a first SLSM circuit SLSM1 210. C2C1 206 also includes a first SLSM monitor SLSMM1 212 for monitoring SLSM1 210. Die 2 204 includes a second C2C serial transceiver C2C2 214 coupled to a second physical layer circuit PHY2 216. C2C2 214 includes a second SLSM circuit SLSM2 218. C2C2 214 includes a second SLSM monitor SLSMM2 220 for monitoring SLSM2 218. In some aspects, SLSM1 210 and SLSM2 218 can be PCIE LTSSMs, and SLSMM1 212 and SLSMM2 220 can be LTSSM monitors. In some aspects, the C2C link can be the same or similar to PCIE, and the examples disclosed herein can reflect this similarity, but note that the same principles can be applied to C2C links of other protocols.

[0037] Die 1 202 and die 2 204 communicate with each other via a C2C link using conductive traces 222 and 224 on or embedded within the substrate 200. The conductive traces 222 and 224 provide an electrical connection between PHY1 212 and PHY2 220. The C2C link itself is not visible at the chip package and is entirely contained within the substrate. The SLSM monitors are used to provide link status, link speed, and time spent in each state for both sides of the link partner using only 2 general purpose input / output ports (GPIOs). In Figure 2 In the example, SLSMM1 212 outputs this information on GPIO1 226, and SLSMM2 220 outputs this information on GPIO2 228. This feature is critical for link startup, link debug, and link optimization.

[0038] The respective SLSM circuits maintain the C2C link training status and current state of their respective die. For simplicity of the description, the operation of only one of the SLSM monitors will be described, but note that the other SLSM monitor will perform the same functions for the other die.

[0039] Referring to C2C1 206 on die 1 202, SLSM1 210 provides the current link state to SLSMM1 212. SLSMM1 212 also knows the core clock rate of die 1 202, core1 clk rate. Accordingly, SLSMM1 212 maintains a running list of the state in which SLSM1 210 is in and the length of time SLSM1 210 is in each state in terms of the number of core clock cycles. Upon request, SLSMM1 212 can provide this information to entities external to die 1 202, e.g., via GPIO1 226. Likewise, SLSMM2 220 maintains the same kind of information about SLSM2 218 and can provide this information to entities external to die 2 204 via GPIO2 228. In some aspects, this information can be provided in the unique form disclosed in more detail below, but other formats for providing this information are within the scope of the present disclosure and aspects thereof.

[0040] In this way, the state of the C2C link between die 1 202 and die 2 204 can be made visible for monitoring and / or analysis external to die 1 202 and die 2 204, such as by logic analyzer 230 (or other type of C2C link monitoring equipment) that analyzes the state of SLSM1 210 and SLSM2 218 by processing serial output data 232 received from GPIO1 226 and serial output data 234 received from GPIO2 228, respectively. In some aspects, logic analyzer 230 can be part of another die mounted to substrate 200. In some aspects, logic analyzer 230 can be a piece of benchtop test equipment electrically connected to substrate 200 for testing and debugging purposes only. These examples are illustrative and not limiting. Other implementations of logic analyzer 230 are contemplated by the present disclosure.

[0041] Analysis of serial output data 232 from SLSMM1 212 and serial output data 234 from SLSMM2 220 allows logic analyzer 230 to monitor and debug both partners of the embedded C2C serial link simultaneously (e.g., one partner per GPIO) and in real-time or near real-time. For clarity, Figure 2 Details regarding triggering SLSMM1 212 or SLSMM2 220 to output serial output data 232 and serial output data 234, respectively, are not shown, but in some aspects, these serial output data can be output in response to an external request (e.g., from logic analyzer 230) or in response to an internal trigger (e.g., once it is available, in which case logic analyzer 230 can monitor for a sequence of bits indicating the start of the serial output data sequence). Other trigger modes are also contemplated by the present disclosure.

[0042] Figure 3 Example SLSM monitors in accordance with aspects of the present disclosure are illustrated in greater detail. In Figure 3 In the example illustrated in FIG. 2, the SLSM monitor SLSMM1 212 is divided into two clock domains, a core clock domain 300 and a serial clock domain 302. An asynchronous first-in-first-out (FIFO) buffer 304 operates in both clock domains. The primary function of the SLSMM1 212 is to monitor the SLSM of the C2C link (e.g., SLSM1 210 in FIG. 1) and output the state along with the number of cycles spent in each state to a GPIO (e.g., GPIO 226) in a serial fashion. In some aspects, the SLSMM1 212 includes a FIFO control block 306 for controlling the asynchronous FIFO 304. The FIFO control block 306 manages a write pointer 308 and a read pointer 310 as well as a write control signal (Wr) and a read control signal (Rd). A control block 312 controls the SLSMM1 212. Figure 2

[0043] core clock domain. In some aspects, the core clock domain 300 stores information indicating a current state of the SLSM (SLSM state) 314 and information indicating a core clock rate 316, and tracks the length of time the SLSM remains in the current state using a counter 318, for example. In some aspects, the core clock domain 300 begins monitoring when enabled (e.g., via a control signal received by the control block 312), and at least one of the following conditions is met: the SLSM state changes to a non-zero value; or the SLSM state equals a particular state and the logic is configured to trigger at the particular state. In some aspects, once triggered, the core clock domain 300 counts the number of core clock cycles spent in the current state. In some aspects, the core clock domain stores the core clock rate, the current SLSM state, and the number of cycles spent in the state to the asynchronous FIFO 304. In some aspects, the asynchronous FIFO 304 is designed to accommodate SLSM state changes for typical link training scenarios. In the event of SLSM state changes that can cause the asynchronous FIFO 304 to fill up more frequently, a condition indicating the error condition will be stored to be carried out in the serial output.

[0044] ​Asynchronous FIFO. In some aspects, the asynchronous FIFO 304 writes data at the core clock frequency. In some aspects, the asynchronous FIFO 304 provides a "FIFO full" flag to the write control logic to avoid overwriting valid data. In some aspects, the asynchronous FIFO 304 reads data when valid data is available at the serial clock frequency. In some aspects, the asynchronous FIFO 304 provides a "FIFO empty" flag to the read control logic to indicate availability of valid data. In some aspects, the read data is provided to the serializer 320, which sends the valid data out in serial fashion via the GPIO.

[0045] Serial clock domain. In some aspects, the serial clock domain 302 reads valid data from the asynchronous FIFO 304 (when available) and outputs the valid data via the GPIO 226. In some aspects, the data read from the asynchronous FIFO 304 includes SLSM state and number of cycles in that state. In some aspects, the serializer 320 serializes the data at the serial clock frequency. In some aspects, for example, in the case of a C2C based on PCIE and the SLSM is a PCIE LTSSM, the serial clock domain 302 transmits the serial data as 32 bits to the GPIO in the following format:

[0046] Bit 31 30:28 27:26 25:20 19:0 Field Valid PCIE gen rate Status LTSSM state / substate Core clock cycle count

[0047] where the core clock cycle count refers to the number of core clock cycles spent in each LTSSM state or sub-state. In some aspects, when the SLSM or core clock rate remains constant for a long duration, the logic will force a serial output and the clock cycle count is set to the maximum configurable value and continue counting using the next entry in the asynchronous FIFO 304. Some examples of serial output are shown in Figures 4A-4C

[0048] Figure 4A Figure 4B Figure 4C are each a diagram illustrating an example SLSM monitor serial output using an example format for presenting SLSM state machine information in serial fashion in accordance with aspects of the present application. In the example shown in Figures 4A-4C the C2C link is based on PCIE and uses a PCIE LTSSM state machine, but the same principles can be applied to other C2C serial protocols. Figures 4A-4C The counter values shown are illustrative and not limiting; actual counter values will typically be higher values than the examples shown in Figures 4A-4C

[0049] Figure 4A ​​​​An example SLSM monitor serial output indicating example LTSSM state transitions at gen 1 speed is shown. The bits of the core_clk_rate field = {000}, which indicates a core clock rate of 62.5 MHz. The bits of the status field = {00}, which indicates OK (e.g., no error or overflow occurred). The bits of the ltssm_state field = {000001}, which indicates the Detect.Active substate. The bits of the core_clk_cycles field = {00000000000000011101}, which indicates 29 cycles.

[0050] Figure 4B An example SLSM monitor serial output is shown for a clock rate transition to gen 4 speed. The bits of the core_clk_rate field = {011}, which indicates a core clock rate of 500 MHz. The bits of the status field = {00}, which indicates OK (e.g., no error or overflow occurred). The bits of the ltssm_state field = {001101}, which indicates the Recovery.Speed substate. The bits of the core_clk_cycles field = {00000000000111011100}, which indicates 476 cycles.

[0051] Figure 4C An example SLSM monitor serial output is shown for an overflow condition. When an overflow condition occurs, the state counter will be forced to 0. The bits of the core_clk_rate field = {011}, which indicates a core clock rate of 500 MHz. The bits of the status field = {01}, which indicates that an overflow occurred. The bits of the ltssm_state field = {001101}, which indicates the Recovery.Speed substate. Due to the overflow condition, the bits of the core_clk_cycles field are all set to zero.

[0052] In some aspects, the serial output can be captured with analyzers on both link partners. This data can then be post-processed to provide visibility into link initialization and training. Several examples of post-processed serial output for both link partners (partner 0 and partner 1) are shown in the following tables. These examples are for illustration only and are intended to show example LTSSM state or substate transitions seen from each end of a PCIE-based example C2C link, but the same principles can be applied to C2C links and state machines using protocols other than PCIE. The time values are chosen to be illustrative and do not necessarily reflect real link training values.

[0053] Table 2 shows the post-processed output of a first example of monitoring a C2C link between partner 0 and partner 1. Table 2 shows an example where each partner is able to establish a unidirectional link at a first link speed, Gen 1, and then increase the link speed to a second link speed, Gen 4. The time taken for each of these steps (i.e. the number of clock cycles the link remains in each respective state or sub-state) is approximately the same for both partners.

[0054] Table 2

[0055]

[0056]

[0057] Of particular note is the time taken at Gen 1 speed (state #3 in Table 2) before the link is able to change to Gen 4 speed (state #4 in Table 2). In the example above, the link stays at Gen 1 for approximately 200 clock cycles before transitioning to Gen 4.

[0058] Table 3 shows the post-processed output of a second example of monitoring a C2C link involving two partners.

[0059] Table 3

[0060]

[0061] In this example, partner 0 stays at Gen 1 speed for 4325 clock cycles before transitioning to Gen 4 speed (state #3 in Table 3), while partner 1 is able to change to Gen 4 speed after the usual 200 clock cycles or so. The time taken to change to Gen 4 speed is longer, indicating that the phase-locked loop (PLL) of partner 0 takes longer to lock to the new speed. With this knowledge, the developer or debugger can determine that the PLL control registers should be adjusted to speed up the locking process. Without this knowledge, the developer or debugger can have little indication that optimisation can be made during link establishment.

[0062] Table 4 shows the post-processed output of a third example of monitoring a C2C link involving two partners.

[0063] Table 4

[0064]

[0065] In this example, partner 0 remained at Gen 1 speed for 4325 clock cycles before transitioning to Gen4 speed (state #3 in Table 4), while partner 1 was able to change to Gen4 speed after about the usual 200 clock cycles, again indicating that the PLL control register on partner 0 should be adjusted to speed up the lock process. However, in addition, the above table indicates that for receiver equalization phase 3 (state #8 in Table 4), partner 1 took much longer than partner 0. This indicates that the default transmit equalization settings on partner 0 were not optimal for partner 1’s signal reception.

[0066] Figure 5 is a flow diagram illustrating portions of an example process 500 associated with an on-die C2C link state monitor, in accordance with aspects of the present disclosure. In some implementations, Figure 5 One or more process blocks of the process 500 can be performed by a C2C link circuit (e.g., C2C serial transceiver 206, SLSMM1 212). In some implementations, Figure 5 One or more process blocks of the process 500 can be performed by another device or a group of devices separate from or including the C2C link circuit. Additionally or alternatively, Figure 5 One or more process blocks of the process 500 can be performed by one or more components of the device, such as a processor, a memory, a transceiver, any or all of which can be means for performing the operations of the process 500.

[0067] As shown in Figure 5 the process 500 can include detecting a change in a link state of a C2C link from a first link state to a second link state, at block 502. As changes in the link state of a C2C serial link are reflected by the SLSM, the change in the link state corresponds to a change in the SLSM state. In some aspects, the means for performing the operations of block 502 can include an on-die C2C link circuit, such as the C2C serial transceiver 206, for communicating via the C2C link in accordance with a C2C protocol. For example, the SLSMM1 212 can receive a notification of a new SLSM state from the SLSM1 210. In some aspects, detecting the change in the link state of the C2C link includes detecting a change in a link state of a Peripheral Component Interconnect Express (PCIE) protocol link.

[0068] As shown in Figure 5Further, process 500 can include, at block 504, storing the link state change information in a first-in-first-out (FIFO) buffer on the die. Means for performing the operation of block 504 can include C2C serial transceiver 206. For example, SLSMM1 212 can receive the SLSM state from SLSM1 210 and, before storing it in SLSM state 314, can create an entry in asynchronous FIFO 304 that includes the previous value in SLSM state 314 and the current value in counter 318. Once this information is stored into asynchronous FIFO 304 (e.g., by FIFO control block 306), write pointer 308 is updated, counter 318 is cleared, and the received SLSM state is stored in SLSM state 314. Thus, in some aspects, the link state change information can include information indicating the first link state, a duration of time that the C2C link was in the first link state, and a speed of the C2C link in the first link state.

[0069] In some aspects, storing information indicating the first link state includes storing information indicating a PCIE LTSSM state or substate. In some aspects, storing information indicating the duration of time that the C2C link was in the first link state includes storing information indicating a number of clock cycles that the C2C link was in the first link state.

[0070] As Figure 5 Further, process 500 can include, at block 506, detecting a request for the link state change information. Means for performing the operation of block 506 can include SLSMM1 212. For example, such a request can be received by control block 312.

[0071] As Figure 5 Further, process 500 can include, at block 508, retrieving the link state change information from the FIFO buffer. Means for performing the operation of block 508 can include SLSMM1 212. For example, control block 312 can trigger FIFO control block 306 to instruct asynchronous FIFO 304 to transfer the oldest entry (pointed to by read pointer 310) to serializer 320 and update read pointer 310.

[0072] As Figure 5 Further, process 500 can include, at block 510, sending the link state change information to the output pin in a serial manner. Means for performing the operation of block 510 can include SLSMM1 212. For example, then, serializer 320 can output the state change information received from asynchronous FIFO 304 via GPIO 226 in a serial manner.

[0073] In some aspects, sending the link state change information to the output pin in a serial fashion includes sending at least one of: one or more bits indicating that the serial output is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C link is in the link state.

[0074] Process 500 can include additional implementations, such as any single implementation or any combination of Figure 5 The example blocks of process 500 are shown in a particular order, but in some implementations, process 500 can include more blocks, fewer blocks, different blocks, or blocks arranged in a different order than those depicted in Figure 5 In some implementations, process 500 can include more blocks, fewer blocks, different blocks, or blocks arranged in a different order than those depicted in FIG. 5. Additionally or alternatively, two or more of the blocks of process 500 can be performed in parallel.

[0075] The on-die SLSM monitor disclosed herein has a number of technical advantages. For example, in a traditional multi-chip module, the link partners are inaccessible because they are embedded in the substrate, but the on-die SLSM monitor provides the ability to monitor link transitions after post-processing of the serialized state data.

[0076] Because the C2C link operates at a very high rate, it is not possible to simply provide an electrical connection directly to the conductive traces on which the link operates without unacceptable degradation of link performance due to the additional impedance, inductance, and noise associated with the electrical connection to an observable package pin. Even assuming that these technical challenges can be overcome, there remains the problem that analysis of the C2C link signal so obtained would require an expensive protocol analyzer.

[0077] In contrast, the on-die SLSM monitor disclosed herein pre-processes the link data to provide real-time status updates, which occur at a rate that is much lower than the data rate of the C2C link itself. Thus, this information can be provided by the SLSM monitor via a GPIO pin or other output pin at a lower clock rate, i.e., which can be easily accommodated by an observable package pin. Because the SLSM monitor disclosed herein provides an overview of the activity on the link, e.g., identifying the states into which the link enters and the length of time that the link stays in each state, this eliminates a portion of the post-processing that would otherwise need to be performed by an external protocol analyzer, simplifying analysis of the data output by the SLSM monitor on the GPIO pin.

[0078] Further, the on-die SLSM monitor provides the ability to monitor SLSM status and C2C link speed simultaneously across two link partners using only two GPIOs in a serial fashion across the C2C link. Since SLSM status updates are provided using a single GPIO on each side as compared to the entire bus, this saves valuable space on the substrate that would otherwise be occupied by signal traces.

[0079] Because there can be an SLSM monitor on each end of a C2C link, both sides of a C2C link can be monitored and debugged simultaneously, which can result in significant test time reduction. According to the same principle, multiple C2C links can be monitored and debugged simultaneously, further reducing test time. Alternatively, the same GPIO pin on the same dielet can be used to monitor a set of C2C links from one dielet to one or more other dielets in sequence. For example, a GPIO pin on dielet A can be used to monitor a link from dielet A to dielet B, then the same GPIO pin on dielet A can be used to monitor a link from dielet A to dielet C, and so on. Alternatively, one GPIO pin on dielet A can be used to monitor a C2C link with dielet B, while another GPIO pin on dielet A can be used to monitor a C2C link with dielet C, and so on.

[0080] Another advantage of the on-die SLSM monitor disclosed herein is that it is able to capture link status information in real time. Conventional methods such as monitoring control and status registers (CSRs) or determining statistics through firmware do not allow for real-time monitoring. Real-time monitoring is critical for debugging transient events that affect platform design as well as power distribution network (PDN) and package characteristics.

[0081] For example, in some aspects, the time spent in each SLSM state provides a quantifiable means to fine-tune various CSR registers of the C2C PHY. In some aspects, adaptive feedback to the PHY optimization loop is used to improve initialization and startup time in real time. In some aspects, such adaptive feedback is deployed as a self-correcting feature over the product life cycle. Further, improving startup time and latency has a direct impact on the power-performance metric, which can be a key product differentiator. Optimization of link initialization based on information provided by the on-die SLSM monitor is useful in applications with tight operational time (such as autonomous driving link-up, where a vehicle needs to be operational within a few seconds), as well as in critical cloud infrastructure solutions. Another benefit to customers is that with the on-die SLSM monitor disclosed herein, the LTSMM status can be a monitor on the customer premises during startup using an inexpensive logic analyzer, for example, without the need to maintain an engineering-intensive firmware image for performing the logic analyzer functionality.

[0082] In the detailed description above, different features were grouped into sections. Such grouping is not intended to limit the examples to the described features, as each feature can be combined with any other feature or features. It is not intended that the claimed examples rely on any particular aspect of the detailed description, but rather, the examples are intended to cover all examples falling within the scope of the claims. It is further noted that the claims can be drafted to exclude any features or aspects, or to include features or aspects that are not recited in the detailed description. Thus, the claims can be drafted to include features or aspects that are not recited in the detailed description, or to exclude features or aspects recited in the detailed description.

[0083] Particular example embodiments are described in the following numbered clauses:

[0084] Clause 1. A method of on-die monitoring of C2C serial link state, the method comprising: detecting a change in link state of the C2C serial link from a first link state to a second link state; storing link state change information in an on-die FIFO buffer, the link state change information comprising information indicating the first link state, a duration of time that the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state; detecting a request for link state change information; retrieving link state change information from the FIFO buffer; and sending the link state change information to an output pin in a serial fashion.

[0085] Clause 2. The method of clause 1, wherein storing information indicating the duration of time that the C2C serial link was in the first link state comprises storing information indicating a number of clock cycles that the C2C serial link was in the first link state.

[0086] Clause 3. The method of any of clauses 1-2, wherein sending the link state change information to the output pin comprises sending the link state change information to a GPIO pin.

[0087] Clause 4. The method of any of clauses 1-3, wherein sending the link state change information to the output pin in a serial fashion comprises sending at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link was in the link state.

[0088] Clause 5. The method of any of clauses 1-4, wherein detecting the change in link state of the C2C serial link comprises detecting the change in link state of a PCIE protocol link, and wherein storing information indicating the first link state comprises storing information indicating a PCIE LTSSM state or substate.

[0089] Clause 6. An apparatus for on-die monitoring of C2C serial link training status and link state, the apparatus comprising: a die comprising: a C2C serial link circuit comprising: a physical layer configured to send and receive data via a C2C serial link according to a C2C communication protocol; an SLSM configured to perform link training of the C2C serial link; and an SLSM monitor coupled to the SLSM and configured to: detect a change in link state of the C2C serial link from a first link state to a second link state; store link state change information in an on-die FIFO buffer, the link state change information comprising information indicating the first link state, a duration of time that the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state; detect a request for link state change information; retrieve link state change information from the FIFO buffer; and send the link state change information to an output pin of the die in a serial fashion.

[0090] Clause 7. The apparatus of clause 6, wherein, to store information indicating the duration of time that the C2C serial link was in the first link state, the SLSM monitor is configured to store information indicating a number of clock cycles that the C2C serial link was in the first link state.

[0091] Clause 8. The apparatus of any of clauses 6-7, wherein, to send the link state change information to the output pin of the die, the SLSM monitor is configured to send the link state change information to a GPIO pin.

[0092] Clause 9. The apparatus of any of Clauses 6-8, wherein to transmit the link state change information to the output pin in a serial fashion, the SLSM monitor is configured to transmit at least one of: one or more bits indicating the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time the C2C serial link was in the link state.

[0093] Clause 10. The apparatus of any of Clauses 6-9, wherein the C2C communication protocol comprises a PCIE protocol, and wherein the serial link state machine comprises a PCIE LTSSM, and wherein the information indicating the first link state comprises information indicating a LTSSM state or substate.

[0094] Clause 11. An apparatus comprising: means for detecting a change in a link state of a C2C serial link from a first link state to a second link state; means for storing link state change information in a die on FIFO buffer, the link state change information comprising information indicating the first link state, a duration of time the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state; means for detecting a request for link state change information; means for retrieving link state change information from the FIFO buffer; and means for transmitting the link state change information to an output pin in a serial fashion.

[0095] Clause 12. The apparatus of Clause 11, wherein the means for storing the information indicating the duration of time the C2C serial link was in the first link state comprises means for storing information indicating a number of clock cycles the C2C serial link was in the first link state.

[0096] Clause 13. The apparatus of any of Clauses 11-12, wherein the means for transmitting the link state change information to the output pin comprises means for transmitting the link state change information to a GPIO pin.

[0097] Clause 14. The apparatus of any of clauses 11-13, wherein the means for sending the link state change information to the output pin in a serial fashion comprises means for sending at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link was in the link state.

[0098] Clause 15. The apparatus of any of clauses 11-14, wherein detecting the change in the link state of the C2C serial link comprises detecting the change in the link state of a PCIE protocol link, and wherein storing information indicating the first link state comprises storing information indicating a PCIE LTSSM state or substate.

[0099] Clause 16. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by an apparatus, cause the apparatus to: detect a change in a link state of a C2C serial link from a first link state to a second link state; store link state change information in an on-die FIFO buffer, the link state change information comprising information indicating the first link state, a duration of time that the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state; detect a request for link state change information; retrieve link state change information from the FIFO buffer; and send the link state change information to an output pin in a serial fashion.

[0100] Clause 17. The non-transitory computer-readable medium of clause 16, wherein the computer-executable instructions that, when executed by the apparatus, cause the apparatus to store information indicating the duration of time that the C2C serial link was in the first link state comprise computer-executable instructions that, when executed by the apparatus, cause the apparatus to store information indicating a number of clock cycles that the C2C serial link was in the first link state.

[0101] Clause 18. The non-transitory computer-readable medium of any of clauses 16-17, wherein the computer-executable instructions that, when executed by the apparatus, cause the apparatus to send the link state change information to the output pin comprise computer-executable instructions that, when executed by the apparatus, cause the apparatus to send the link state change information to a GPIO pin.

[0102] Clause 19. The non-transitory computer-readable medium of any of clauses 16-18, wherein the computer-executable instructions that, when executed by the device, cause the device to send the link state change information to the output pin in a serial fashion comprise computer-executable instructions that, when executed by the device, cause the device to send at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link is in the link state.

[0103] Clause 20. The non-transitory computer-readable medium of any of clauses 16-19, wherein detecting the change in link state of the C2C serial link comprises detecting the change in link state of a PCIE protocol link, and wherein storing information indicating the first link state comprises storing information indicating a PCIE LTSSM state or substate.

[0104] Clause 21. An apparatus comprising: a memory; a transceiver; and a processor communicatively coupled to the memory and the transceiver, the memory, the transceiver, and the processor configured to perform the method of any of clauses 1-5.

[0105] Clause 22. An apparatus comprising means for performing the method of any of clauses 1-5.

[0106] Clause 23. A non-transitory computer-readable medium storing computer-executable instructions, the computer-executable comprising at least one instruction for causing a computer or processor to perform the method of any of clauses 1-5.

[0107] Those skilled in the art will understand that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0108] Moreover, those skilled in the art will appreciate that the functions described herein can be implemented using software functioning in connection with a programmable computer system; programmable logic components, discrete hardware components, or any combination thereof. As used herein the term "computer" includes one or more tangible computer hardware components, such as those based on semiconductors, microprocessors, microcomputers, microcontrollers, application- specific integrated circuits, etc. The term "software" includes any instructions executed on a computer, such as those that cause a computer to carry out an algorithm. Computer software is a tangible article of manufacture that embodies a set of instructions that, when executed by a computer, cause the computer to carry out an algorithm. Memories or memory elements of the computer can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, optical media, tape, or another suitable type of memory used for storage of instructions for execution by a processor or processors of a suitable computer system. A "computer-readable medium" can include one or more memory elements.

[0109] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0110] The methods, sequences, and / or algorithms described in connection with the aspects disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal (e.g., an UE). In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0111] In one or more example aspects, functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0112] While the foregoing disclosure shows illustrative aspects of the present disclosure, it should be noted that various changes and modifications could be made therein without departing from the scope of the present disclosure as defined by the appended claims. Moreover, the functions, steps and / or actions of the methods according to the aspects of the disclosure described herein need not be performed in any particular order. Furthermore, although elements of the present disclosure can be described or claimed in singular form, plural forms can also be considered unless explicitly stated otherwise.

Claims

1. A method for on-chip monitoring of the status of a chip-to-chip C2C serial link, the method comprising: At the C2C serial link state machine monitoring circuit on the die used to monitor the serial link state machine for communication via the C2C serial link according to the C2C protocol: Detect the change in the link state of the C2C serial link from the first link state to the second link state; The link state change information is stored in the on-die first-in-first-out FIFO buffer. The link state change information includes information indicating the first link state, the duration of the C2C serial link in the first link state, and the speed of the C2C serial link in the first link state. The information storing the duration of the C2C serial link in the first link state includes information storing the number of clock cycles of the C2C serial link in the first link state. The system detects that the speed of the C2C serial link in the first link state exceeds the duration of the overflow condition and sets the information indicating the number of clock cycles in the first link state of the C2C serial link to a configurable value. Detect requests for link state change information; Retrieve link state change information from the FIFO buffer; as well as The link status change information is sent serially to the output pin of the die.

2. The method of claim 1, wherein sending the link state change information to the output pin includes sending the link state change information to a general purpose input / output (GPIO) pin.

3. The method of claim 1, wherein sending the link state change information to the output pin serially includes sending at least one of the following: One or more bits indicating that the link state change information is valid; One or more bits indicating the core clock rate; One or more bits indicating the status of the link state change information; One or more bits indicating the link status; or One or more bits indicating the duration of the C2C serial link being in the link state.

4. The method of claim 1, wherein detecting the change in the link state of the C2C serial link includes detecting the change in the link state of the peripheral component fast interconnect (PCIE) protocol link, and wherein storing information indicating the first link state includes storing information indicating the PCIE link training status and the state machine LTSSM state or sub-state.

5. An on-die monitoring device for chip-to-chip C2C serial link training status and link status, the device comprising: The die includes: The C2C serial link circuit includes: The physical layer is configured to send and receive data via a C2C serial link according to the C2C communication protocol; A serial link state machine (SLSM) configured to perform link training on the C2C serial link; and an SLSM monitor coupled to the SLSM and configured to: detect a change in a link state of the C2C serial link from a first link state to a second link state; store link state change information in an on-die first-in-first-out (FIFO) buffer, the link state change information including information indicating the first link state, a duration of time that the C2C serial link was in the first link state, and a speed of the C2C serial link in the first link state, wherein the information indicating the duration of time that the C2C serial link was in the first link state includes information indicating a number of clock cycles that the C2C serial link was in the first link state; detect that the speed of the C2C serial link in the first link state exceeds the duration of time and set the information indicating a number of clock cycles that the C2C serial link was in the first link state to a configurable value; detect a request for link state change information; retrieve link state change information from the FIFO buffer; and send the link state change information to an output pin of the die in a serial fashion.

6. The apparatus of claim 5, wherein, To send the link state change information to the output pin of the die, the SLSM monitor is configured to send the link state change information to a general purpose input / output (GPIO) pin.

7. The apparatus of claim 5, wherein, To send the link state change information to the output pin in a serial fashion, the SLSM monitor is configured to send at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link was in the link state.

8. The apparatus of claim 5, wherein the C2C communication protocol comprises a peripheral component interconnect express (PCIE) protocol, and wherein the serial link state machine comprises a PCIE link training status and state machine (LTSSM), and wherein the information indicating the first link state includes information indicating an LTSSM state or substate.

9. An apparatus comprising: at a C2C serial link state machine monitor circuit on a die for monitoring a serial link state machine for communicating via a C2C serial link according to a C2C protocol: means for detecting a change in a link state of the C2C serial link from a first link state to a second link state; ​ means for storing link state change information in a first-in-first-out FIFO buffer on a die, the link state change information including information indicating the first link state, a duration of time that the C2C serial link is in the first link state, and a speed of the C2C serial link in the first link state, wherein the information indicating the duration of time that the C2C serial link is in the first link state includes information indicating a number of clock cycles that the C2C serial link is in the first link state; means for detecting that the speed of the C2C serial link in the first link state exceeds the duration of time and setting the information indicating a number of clock cycles that the C2C serial link is in the first link state to a configurable value; means for detecting a request for link state change information; means for retrieving link state change information from the FIFO buffer; and means for sending the link state change information to an output pin of the die in a serial fashion.

10. The apparatus of claim 9, wherein the means for sending the link state change information to the output pin comprises means for sending the link state change information to a general purpose input / output (GPIO) pin.

11. The apparatus of claim 9, wherein the means for sending the link state change information to the output pin in a serial fashion comprises means for sending at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link is in the link state.

12. The apparatus of claim 9, wherein detecting the change in link state of the C2C serial link comprises detecting the change in link state of a peripheral component interconnect express (PCIE) protocol link, and wherein storing information indicating the first link state comprises storing information indicating a PCIE link training status and state machine (LTSSM) state or substate.

13. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by an apparatus, cause the apparatus to: at a C2C serial link state machine monitoring circuit on a die for monitoring a serial link state machine for communicating via a C2C serial link according to a C2C protocol: detect a change in link state of the C2C serial link from a first link state to a second link state; ​ storing link state change information in an on-die first-in-first-out (FIFO) buffer, the link state change information including information indicating the first link state, a duration of time that the C2C serial link is in the first link state, and a speed of the C2C serial link in the first link state, wherein the information indicating the duration of time that the C2C serial link is in the first link state includes information indicating a number of clock cycles that the C2C serial link is in the first link state; detecting that the speed of the C2C serial link in the first link state exceeds the duration of time and setting the information indicating a number of clock cycles that the C2C serial link is in the first link state to a configurable value; detecting a request for link state change information; retrieving link state change information from the FIFO buffer; and sending the link state change information to an output pin in a serial fashion.

14. The non-transitory computer-readable medium of claim 13, wherein the computer-executable instructions that, when executed by the device, cause the device to send the link state change information to the output pin comprise computer-executable instructions that, when executed by the device, cause the device to send the link state change information to a general purpose input / output (GPIO) pin.

15. The non-transitory computer-readable medium of claim 13, wherein the computer-executable instructions that, when executed by the device, cause the device to send the link state change information to the output pin in a serial fashion comprise computer-executable instructions that, when executed by the device, cause the device to send at least one of: one or more bits indicating that the link state change information is valid; one or more bits indicating a core clock rate; one or more bits indicating a status of the link state change information; one or more bits indicating a link state; or one or more bits indicating a duration of time that the C2C serial link is in the link state.

16. The non-transitory computer-readable medium of claim 13, wherein detecting the change in link state of the C2C serial link comprises detecting the change in link state of a peripheral component interconnect express (PCIE) protocol link, and wherein storing information indicating the first link state comprises storing information indicating a PCIE link training status and a link training state selection and status machine (LTSSM) state or substate. ​

Images