Tax message transmission in a multi-slot interconnect layer flit

Abstract

A link-layer control message is generated and enclosed in a flit, which is to be sent to a device via a serial data link. The flits sent over the data link must contain multiple slots. Control messages can, in some aspects, include a virus alert message, a poison alert message, a loan repayment message, and receipts.

Description

AREA

[0001] The present disclosure relates generally to the field of computer development and more specifically to software development, which involves the coordination of interdependent constrained systems. BACKGROUND

[0002] Advances in semiconductor processing and logic design have enabled an increase in the amount of logic that can reside on integrated circuit devices. As a consequence, computer system designs have evolved from a single or multiple integrated circuits within a system to multiple cores, multiple hardware subprocessors, and multiple logic processors, which can reside on single integrated circuits, as well as other interfaces integrated within such processors. A processor or integrated circuit typically comprises a single physical processor chip, which can include any number of cores, hardware subprocessors, logic processors, interfaces, memories, control unit nodes, and so on.

[0003] As a result of the increased ability to pack more processing power into smaller enclosures, smaller computing devices have gained popularity. Smartphones, tablets, ultrathin notebooks, and other consumer devices have shown exponential growth. However, these smaller devices rely on servers for both data storage and complex processing that exceeds their form factor. Accordingly, demand in the high-performance computing market (i.e., server space) has also increased. For example, modern servers typically contain not just a single multi-core processor, but multiple physical processors (also known as multiple sockets) to increase computing power. However, as processing power increases along with the number of devices in a computing system, communication between sockets and other devices becomes more critical.

[0004] In fact, interconnects have evolved from more traditional multidrop buses, which primarily handled electrical communications, into fully developed interconnect architectures that facilitate high-speed communication. Unfortunately, with the requirement for future processors to consume even higher data rates, the corresponding demand is now focused on the capabilities of existing interconnect architectures.

[0005] US 6,628,615 B1 discloses a system and method for transmitting messages between nodes of a packet-switched communication network, wherein each message has a defined message type and includes message content. The system comprises one or more second-level channel interface devices connected to a first node to track information related to the bidirectional communication of packets over a communication channel established between the first and a second network node; a device for receiving packets associated with messages from the first node and for generating message fragments (flits) associated with the messages for communication over the channel based on the message content of the received message packets;A device for receiving message fragments (flits) associated with messages communicated by a second node and received over the channel, and for generating the corresponding message packet content for storage at the first node; and one or more first-level channel interface devices associated with one or more second-level channel interface devices and connected to a network switch at each first and second node to communicate flits to and from a respective first and second node over the channel, the communication channel established between the first and second network node comprising a first-level and a second-level channel selected according to the message content.

[0006] US 7,583,600 B1 discloses a connector for a PCI Express connection. The connector can include a scheduler that can be operated to determine the next transmission time for management packets, and a window determiner that can be operated to determine a transmission window based on the next transmission time for management packets. The connector can also include an inserter that can be operated to examine a data packet stream to determine whether a gap occurs within it during the transmission window and, if such a gap occurs, to control the insertion of a management packet into it. This can increase the transmission efficiency for management packets over the connection. BRIEF DESCRIPTION OF THE DRAWINGS Fig.Figure 1 illustrates a simplified block diagram of a system comprising a serial point-to-point intermediate connection to connect I / O devices in a computer system, according to one embodiment. Fig. Figure 2 illustrates a simplified block diagram of a layered protocol stack according to one embodiment. Fig. Figure 3 illustrates an embodiment of a serial point-to-point connection. Fig. Figure 4 illustrates embodiments of potential high-performance interconnect (HPI) system designs. Fig. Figure 5 illustrates an embodiment of a layered protocol stack associated with HPI. Fig. Figure 6 illustrates an example of a multi-slot flitt. Fig.Figure 7 illustrates a representation of an example flit transmitted over an example eight-lane data link. Fig. Figure 8 illustrates a representation of an example flit transmitted over an example eight-lane data link. Fig. Figure 9 illustrates a representation of an exemplary flit transmitted over an exemplary twenty-lane data link. Fig. Figure 10 illustrates a representation of an exemplary virus error tax fluctuation. Fig. Figure 11 illustrates a representation of an exemplary multi-layered flit that includes a diagnostic message. Fig. Figure 12 illustrates a representation of an exemplary poison error tax fluctuation. Fig. Figure 13 illustrates a representation of an example slot message for repaying loans and receipts. Fig.Figure 14 illustrates loan repayment formats for use in the example slot of Fig. 13. Fig. Figure 15 illustrates an embodiment of a block for an exemplary computer system.

[0007] Similar reference numbers and designations in the different drawings indicate similar elements. DETAILED DESCRIPTION

[0008] The following description provides numerous specific details, such as examples of specific processor types and system layouts, specific hardware structures, specific architectural and microarchitecture details, specific register layouts, specific instruction types, specific system components, specific processor pipeline stages, specific intermediate link layers, specific packet / transaction layouts, specific transaction names, specific protocol exchanges, specific link widths, specific implementations and operations, etc., to offer a basic understanding of the present invention. However, it may be clear to those skilled in the art that these specific details are not necessarily required to put the subject matter of this disclosure into practice.In other cases, a well-detailed description of known components or procedures, such as specific and alternative processor architectures, specific logic circuits / codes for described algorithms, specific firmware codes, low-level interconnect operations, specific logic designs, specific manufacturing techniques and materials, specific compiler implementations, specific expressions of algorithms in code, specific shutdown and gate control techniques / logic, and other specific operational details of computer systems, has been avoided in order to prevent the present disclosure from appearing unnecessarily unclear.

[0009] Although the following embodiments, with reference to energy saving, energy efficiency, processing efficiency, etc., can be described in specific integrated circuits, such as computer platforms or microprocessors, other embodiments can be applied to other types of integrated circuits and logic devices. Similar techniques and teachings of embodiments described herein can be applied to other types of circuits or semiconductor devices that can also benefit from such features. For example, the disclosed embodiments are not limited to server computer systems, desktop computer systems, laptops, and Ultrabooks™, but can also be used in other devices, such as handheld devices, smartphones, tablets, other thin notebooks, system-on-a-chip (SoC) devices, and embedded applications.Some examples of handheld devices include mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Similar high-performance intermediary techniques can be applied here to increase performance (or even save energy) in a low-power intermediary. Embedded applications typically include a microcontroller, digital signal processor (DSP), system-on-a-chip, network PC, set-top box, network node, wide area network (WAN) switch, or any other system capable of performing the functions and operations described below. Furthermore, the devices, methods, and systems described here are not limited to physical computing devices but can also relate to software optimizations for energy saving and efficiency.As can be easily seen from the description below, the embodiments of the methods, devices and systems described herein (whether referring to hardware, firmware, software or a combination thereof) can be considered essential for a “green technology” future in balance with performance considerations.

[0010] As computer systems advance, their components become more complex. The interconnect architecture used to couple and communicate between these components has also increased in complexity to ensure bandwidth requirements are met for optimal component operation. Furthermore, different market segments demand different aspects of interconnect architectures to suit their specific needs. For example, servers require higher performance, while the mobile ecosystem may sometimes sacrifice overall performance for energy savings. Nevertheless, a primary goal of most fabrics is to provide the highest possible performance with maximum energy savings. Moreover, a wide variety of interconnects can potentially benefit from the subject matter described here.For example, the Peripheral Component Interconnect (PCI) Express (PCIe) Interconnect Fabric architecture and QuickPath Interconnect (QPI) Fabric architecture, among other examples, can potentially be improved according to one or more of the principles described here, among other examples.

[0011] Fig.Figure 1 illustrates an embodiment of a fabric consisting of point-to-point connections linking a set of components, as shown. The system 100 comprises a processor 105 and a system memory 110 coupled to a control unit node 115. The processor 105 can comprise any processing element, such as a microprocessor, a host processor, an embedded processor, a coprocessor, or another processor. The processor 105 is connected to the control unit node 115 by a front-side bus (FSB) 106. In one embodiment, the FSB 106 is a serial point-to-point interface, as described below. In another embodiment, the interface 106 comprises a serial differential interface architecture compatible with a different interface standard.

[0012] System memory 110 comprises any storage device, such as random access memory (RAM), non-volatile (NV) memory, or other memory accessible by devices in system 100. System memory 110 is coupled to the control unit node 115 via a memory interface 116. Examples of memory interfaces include a dual data rate (DDR) memory interface, a dual-channel DDR memory interface, and a dynamic RAM (DRAM) memory interface.

[0013] In one embodiment, the control unit node 115 may comprise a root node, root complex, or root control unit, as in a PCIe interconnect hierarchy. Examples of a control unit node 115 include a chipset, a memory controller hub (MCH), a northbridge, an interconnect controller hub (ICH), a southbridge, and a root control unit / node. Often, the term chipset refers to two physically separate control unit nodes, e.g., a memory controller node (MCH) coupled to an interconnect controller hub (ICH). It should be noted that current systems often include the MCH integrated with the processor 105, while the control unit 115 communicates with I / O devices in a manner similar to that described below.In some embodiments, peer-to-peer routing may be supported by the root complex 115.

[0014] Here, the control unit node 115 is coupled to a switch / bridge 120 via a serial connection 119. Input / output modules 117 and 121, which can also be referred to as interfaces / ports 117 and 121, can include / implement a layered protocol stack to provide communication between the control unit node 115 and the switch 120. In one embodiment, multiple devices are capable of being coupled to the switch 120.

[0015] The switch / bridge 120 directs packets / messages from the device 125 upstream, i.e., one hierarchy upwards to a root complex, to the control unit node 115, and downstream, i.e., one hierarchy downwards from a root control unit, from the processor 105 or system memory 100 to the device 125. In one embodiment, the switch 120 is referred to as a logical arrangement of several virtual PCI-to-PCI bridge devices.Device 125 comprises any internal or external device or component capable of coupling to an electronic system, such as an I / O device, a network interface controller (NIC), an expansion card, an audio processor, a network processor, a hard disk drive, a storage device, a CD / DVD-ROM drive, a monitor, a printer, a mouse, a keyboard, a router, a portable storage device, a FireWire device, a Universal Serial Bus (USB) device, a scanner, and other input / output devices. In PCIe terminology, such a device is often referred to as an endpoint. Although not specifically shown, Device 125 may include a bridge (e.g., a PCIe-to-PCI / PCI-X bridge) to support legacy or other versions of devices or interconnect fabrics supported by such devices.

[0016] A graphics accelerator 130 can also be coupled to the control unit node 115 via the serial connection 132. In one embodiment, the graphics accelerator 130 is coupled to an MCH, which is coupled to an ICH. The switch 120, and consequently the I / O device 125, is then coupled to the ICH. I / O modules 131 and 118 also implement a layered protocol stack and associated logic to communicate between the graphics accelerator 130 and the control unit node 115. Similar to the MCH discussion above, a graphics control unit or the graphics accelerator 130 itself can be integrated into the processor 105.

[0017] With reference to Fig.Figure 2 illustrates an embodiment of a layered protocol stack. The layered protocol stack 200 can comprise any form of layered communications stack, such as a QPI stack, a PCIe stack, a next-generation high-performance computing interconnect (HPI) stack, or another layered stack. In one embodiment, the protocol stack 200 can comprise a transaction layer 205, a link layer 210, and a physical layer 220. An interface, such as interfaces 117, 118, 121, 122, 126, and 131 in Fig. 1, can be represented as a communication protocol stack 200. The representation as a communication protocol stack can also be referred to as a module or interface that implements / comprises a protocol stack.

[0018] Packets can be used to communicate information between components. Packets can be formed in the transaction layer (205) and the data link layer (210) to carry information from the sending component to the receiving component. As the sent packets pass through the other layers, they are augmented with additional information used for packet handling in those layers. On the receiving side, the reverse process occurs, and packets are transformed from their representation in the physical layer (220) to their representation in the data link layer (210) and finally (for transaction layer packets) into a form that can be processed by the transaction layer (205) of the receiving device.

[0019] In one embodiment, the transaction layer 205 can provide an interface between a device's processing kernel and the intermediate link architecture, such as a data link layer 210 and a physical layer 220. In this respect, a primary responsibility of the transaction layer 205 can include the assembly and disassembly of packets (i.e., transaction layer packets or TLPs). The transaction layer 205 can also manage credit-based flow control for TLPs. In some implementations, split transactions can be used, i.e., transactions with a time-separated request and response, allowing, among other examples, a link to carry other traffic while the destination device gathers data for the response.

[0020] Credit-based flow control can be used to implement virtual channels and networks using the Interconnect Fabric. In one example, a device can offer an initial credit amount for each of the receive buffers in transaction layer 205. An external device at the opposite end of the connection, such as a control unit node 115 in Fig. 1. The system can count the number of credits consumed by each TLP. A transaction can be sent if it does not exceed a credit limit. Upon receiving a response, a credit amount is restored. One example of the advantage of such a credit scheme, among other potential benefits, is that the latency of credit repayment does not impact performance, provided the credit limit is not reached.

[0021] In one embodiment, four transaction address spaces can include a design space, a memory address space, an input / output address space, and a message address space. Memory space transactions comprise one or more read and write requests to transfer data to / from a memory-mapped location. In one embodiment, memory space transactions can use two different address formats, for example, a short address format such as a 32-bit address, or a long address format such as a 64-bit address. Design space transactions can be used to access a design space of different devices connected to the intermediate link. Design space transactions can include read and write requests. Message space transactions (or simply messages) can also be defined to support in-band communication between intermediate link agents.Therefore, in an exemplary embodiment, the transaction layer 205 can assemble package headers / payload information 206.

[0022] A link layer 210, also referred to as the data link layer 210, can act as an intermediate layer between the transaction layer 205 and the physical layer 220. In one embodiment, the responsibility of the data link layer 210 is to provide a reliable mechanism for exchanging transaction layer packets (TLPs) between two components on a link. One side of the data link layer 210 accepts TLPs assembled by the transaction layer 205, applies a packet sequence identifier 211 (i.e., an identification number or packet number), calculates and applies an error detection code (CRC 212), and submits the modified TLPs to the physical layer 220 for transmission across a physical layer to an external device.

[0023] In one example, physical layer 220 comprises a logical subblock 221 and an electrical subblock 222 to physically send a packet to an external device. Here, logical subblock 221 is responsible for the "digital" functions of physical layer 221. In this respect, the logical subblock may include a send section to prepare outgoing information for transmission through physical subblock 222, and a receive section to identify and prepare received information before it is forwarded to the interconnect layer 210.

[0024] The physical block 222 comprises a transmitter and a receiver. The transmitter is supplied with symbols by the logical subblock 221, which the transmitter serializes and forwards to an external device. The receiver is supplied with serialized symbols from an external device and converts the received signals into a bitstream. The bitstream is deserialized and fed back to the logical subblock 221. In one exemplary embodiment, an 8b / 10b transmission code is used, with ten-bit symbols being sent / received. Here, special symbols are used to enclose a packet with frame 223. Additionally, in one example, the receiver also provides a symbol clock, which is recovered from the incoming serial stream.

[0025] Although, as stated above, transaction layer 205, link layer 210, and physical layer 220 are discussed with reference to a specific implementation of a protocol stack (such as a PCIe protocol stack), a layered protocol stack is not so restricted. In fact, any layered protocol stack can be included / implemented and adopt features discussed here. As an example, a port / interface represented as a layered protocol can include: (1) a first layer to assemble packets, i.e., a transaction layer; (2) a second layer to sequence packets, i.e., a link layer; and (3) a third layer to send the packets, i.e., a physical layer. As a specific example, a high-performance intermediate link layer protocol, as described here, is used.

[0026] With reference to next Fig.Figure 3 illustrates an exemplary embodiment of a serial point-to-point fabric. A serial point-to-point link can include any transmission path for sending serial data. In the embodiment shown, a link can include two low-voltage, differentially driven signal pairs: a transmit pair 306 / 311 and a receive pair 312 / 307. Accordingly, the device 305 includes a transmit logic 306 to send data to the device 310 and a receive logic 307 to receive data from the device 310. In other words, two transmit paths, i.e., paths 316 and 317, and two receive paths, i.e., paths 318 and 319, are included in some implementations of a link.

[0027] A transmission path refers to any route for sending data, such as a transmission line, a copper wire, an optical line, a wireless communication channel, an infrared communication link, or any other communication path. A connection between two devices, such as Device 305 and Device 310, is called a link, like Link 315. A link can support one track—each track representing a set of differential signal pairs (one pair for transmitting, one pair for receiving). To scale bandwidth, a link can aggregate multiple tracks, denoted xN, where N is any supported link width, such as 1, 2, 4, 8, 12, 16, 32, 64, or wider.

[0028] A differential pair can refer to two transmission paths, such as lines 316 and 317, for transmitting differential signals. For example, if line 316 switches back and forth from a low voltage level to a high voltage level (i.e., a rising edge), line 317 switches from a high logic level to a low logic level (i.e., a falling edge). Differential signals potentially exhibit better electrical characteristics, such as improved signal integrity (i.e., reduced cross-coupling, voltage overshoot / undershoot, and ringing), among other advantages. This allows for a better timing window, which permits faster transmission frequencies.

[0029] In one embodiment, a novel high-performance interface (HPI) is provided. The HPI can comprise a cache-coherent, connection-based, next-generation interface. For example, an HPI can be used in high-performance computing platforms, such as workstations or servers, including systems where PCIe or another interface protocol is typically used to connect processors, accelerators, I / O devices, and the like. However, the HPI is not limited to these applications. Instead, an HPI can be used in any of the systems or platforms described herein. Furthermore, the individual concepts developed can be applied to other interfaces and platforms, such as PCIe, MIPI, QPI, etc.

[0030] To support multiple devices, in one exemplary implementation the HPI can be instruction set architecture (ISA) agnostic (i.e., the HPI can be implemented in several different devices). In another scenario, the HPI can also be used to connect high-performance I / O devices, not just processors or accelerators. For example, a high-performance PCIe device can be coupled to an HPI via a suitable translation bridge (i.e., HPI to PCIe). Furthermore, the HPI interconnects can be used by many HPI-based devices, such as processors, in a variety of configurations (e.g., stars, rings, nets, etc.). Fig.Figure 4 illustrates exemplary implementations of several potential multi-socket layouts. A dual-socket layout 405, as shown, can include two HPI connections; however, other implementations may use a single HPI connection. For larger topologies, any layout can be used as long as an identifier (ID) can be assigned and some form of virtual path exists, among other additional or complementary features. As shown, a quad-socket layout 410 has an HPI connection from each processor to every other. However, in the eight-socket layout shown in layout 415, not every socket is directly connected to every other socket via an HPI connection. If a virtual path or channel exists between the processors, the layout is supported. A range of supported processors includes 2–32 in a native domain.Higher numbers of processors can be achieved, among other examples, by using multiple domains or other intermediate connections between node control units.

[0031] The HPI architecture comprises a definition of a layered protocol architecture, which in some examples includes protocol layers (coherent, incoherent, and, where applicable, other memory-based protocols), a routing layer, a link layer, and a physical layer with associated I / O logic. Furthermore, the HPI can also include, among other examples, enhancements related to power managers (such as power control units, PCUs), design for test and debugging (DFT), error handling, registers, and security. Fig. Figure 5 illustrates an embodiment of an exemplary HPI-layered protocol stack. In some implementations, at least some of the features shown in Fig.The five illustrated layers are optional. Each layer handles its own level of granularity or its own quantity of information (the protocol layer 505a,b with packets 530, the link layer 510a,b with flits 535, and the physical layer 505a,b with phits 540). It should be noted that in some embodiments, based on the implementation, a packet may comprise partial flits, a single flit, or multiple flits.

[0032] As a first example, the width of a Phit 540 is a 1:1 mapping of the link width to bits (e.g., a 20-bit link width corresponds to a Phit with 20 bits, etc.). Flits can have larger sizes, such as 184, 192, or 200 bits. It's important to note that if a Phit 540 is 20 bits wide and the size of the Flit 535 is 184 bits, then a fraction of Phits 540 is required to send a Flit 535 (e.g., among other examples, 9.2 Phits with 20 bits each to send a 184-bit Flit 535, or 9.6 Phits with 20 bits each to send a 192-bit Flit). It should be noted that the widths of the underlying link can vary at the physical layer. For example, the number of lanes per direction can comprise 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, etc. In one embodiment, the interconnect layer 501a,b can embed several pieces of different transactions in a single flit, and one or more heads (e.g.,1, 2, 3, 4) can be embedded within the flit. In one example, the HPI divides the heads into corresponding slots to allow multiple messages in the flit, each destined for a different node.

[0033] In one embodiment, physical layer 505a,b can be responsible for the rapid transfer of information on the physical medium (electrical or optical, etc.). The physical connection can be point-to-point between two link layer units, such as layers 505a and 505b. Link layer 510a,b can abstract physical layer 505a,b from the upper layers and provides the capability to reliably transfer data (and requests) and manage flow control between two directly connected units. The link layer can also be responsible for virtualizing the physical channel into multiple virtual channels and message classes.The protocol layer 520a,b accesses the link layer 510a,b to map protocol messages into the appropriate message classes and virtual channels before they are passed to the physical layer 505a,b for transfer across the physical links. The link layer 510a,b can support multiple messages, such as request, snoop, response, write-back, and non-coherent data, among others.

[0034] The physical layer 505a,b (or Physical Layer, PHY) of the HPI can be implemented above the electrical layer (i.e., electrical conductors connecting two components), and below the interconnect layer 510a,b, as in Fig.Figure 5 illustrates this. The physical layer and corresponding logic can reside on each agent and connect the interconnect layers on two agents (A and B) that are separated from each other (e.g., on devices on either side of a connection). The local and remote electrical layers are connected by physical media (e.g., wires, conductors, optical, etc.). In one embodiment, the physical layer 505a,b has two main phases: initialization and operation. During initialization, the connection is opaque to the interconnect layer, and signaling can involve a combination of time-controlled states and acknowledgment exchange events. During operation, the connection is transparent to the interconnect layer, and signaling occurs at a single rate, with all traces operating with each other as a single connection.During the operational phase, the physical layer transports flits from agent A to agent B and from agent B to agent A. The link layer abstracts some physical aspects, including media, width, and speed, from the link layers, while flits and control / status of the current layout (e.g., width) are exchanged with the link layer. The initialization phase includes auxiliary phases, such as transmit retrieval and layout. The operational phase also includes auxiliary phases (e.g., link power management states).

[0035] In one embodiment, the link layer 510a,b can be implemented to provide reliable data transfer between two protocol or routing units. The link layer can abstract the physical layer 505a,b from the protocol layer 520a,b and can be responsible for flow control between two protocol agents (A, B), and provide virtual channel services for the protocol layer (message classes) and routing layer (virtual networks). The interface between the protocol layer 520a,b and the link layer 510a,b can typically be at the packet level. In one embodiment, the smallest transfer unit in the link layer is called a flit with a specified number of bits, such as 192 bits or any other nominal value.The 510a,b link layer accesses the 505a,b physical layer to frame the transfer unit of the 505a,b physical layer (phit) into the transfer unit of the 510a,b link layer (flit). Additionally, the 510a,b link layer can be logically divided into two parts: a sender and a receiver. A sender / receiver pair on one layer can be connected to a receiver / sender pair on another layer. Flow control is often performed on both a flit and packet basis. Error detection and correction are also potentially performed on a flit level.

[0036] In one embodiment, the routing layer 515a,b can provide a flexible and distributed method for routing HPI transactions from an origin to a destination. The scheme is flexible because routing algorithms for multiple topologies can be specified by programmable routing tables at each router (in one embodiment, the programming is performed by firmware, software, or a combination thereof). The routing functionality can be distributed; routing can be performed through a series of routing steps, each routing step being defined by a lookup in a table at either the origin, intermediate, or destination router. The lookup at the origin can be used to inject an HPI packet into the HPI fabric. The lookup at an intermediate router can be used to route an HPI packet from an input port to an output port.Looking up a destination port can be used to target the HPI protocol agent. It's worth noting that the routing layer can be thin in some implementations because the routing tables, and therefore the routing algorithms, are not specifically defined by any specification. This allows for flexibility and a wide variety of usage models, including flexible platform architecture topologies, to be defined by the system implementation. Routing layer 515a,b leverages link layer 510a,b to provide for the use of up to three (or more) virtual networks (VNs)—in one example, two non-blocking VNs, VN0 and VN1—with some message classes defined in each virtual network.A shared adaptive virtual network (VNA) can be defined in the link layer, but this adaptive network cannot be directly exposed in routing concepts, since each message class and virtual network may have dedicated resources and guaranteed forward progression, among other features and examples.

[0037] In one embodiment, the HPI can include a coherence protocol layer 520a,b, wherein its supporting agents cache data lines from memory. An agent that wants to cache memory data can use the coherence protocol to read the data line to be loaded into its cache. An agent that wants to modify a data line in its cache can use the coherence protocol to acquire ownership of the line before the data is modified. After modifying a line, an agent can comply with protocol requests to keep it in its cache until it either writes the line back to memory or includes the line in a response to an external request. Finally, an agent can comply with external requests to invalidate a line in its cache. The protocol ensures data coherence by providing rules that all cache agents can follow.It also provides the means for agents without caches to read and write storage data coherently.

[0038] Two states can be enforced to support transactions using the HPI coherence protocol. First, the protocol can maintain data consistency, for example, on a per-address basis, among the data in agent caches and between that data and the data in memory. Informally, data consistency can refer to each valid row of data in an agent's cache representing the most recent value of the data, and data sent in a coherence protocol packet can represent the most recent value of the data at the time it was sent. If no valid copy of the data exists in caches or in transit, the protocol can ensure that the most recent value of the data resides in memory. Second, the protocol can provide well-defined binding points for requests.Binding points for a read can indicate when the data is usable; and for a write, they can indicate when the written data is globally observable and will be loaded on a subsequent read. The protocol can support these binding points for both cacheable and uncacheable (UC) requests in coherent memory space.

[0039] The HPI coherence protocol can also ensure the forward progression of coherence requests made by an agent to an address in the coherent memory space. Transactions can ultimately be fulfilled and deferred for proper system operation. In some implementations, the HPI coherence protocol may lack knowledge of a retry to resolve resource allocation conflicts. Thus, the protocol itself can be defined to have no circular resource dependencies, and implementations can take care in their training to avoid introducing dependencies that lead to blocking. Additionally, the protocol can indicate where trainings are able to provide fair access to protocol resources.

[0040] Logically, in one embodiment, the HPI coherence protocol can comprise three elements: coherence (or cache) agents, home agents, and the HPI interlink fabric that connects the agents. Coherence agents and home agents can work together to achieve data consistency by exchanging messages over the interlink. Link layer 510a,b and its related description can provide the details of the interlink fabric, including how it adheres to the requirements of the coherence protocol, as discussed here. (It should be noted that the division into coherence agents and home agents is for clarity. An education, among other examples, might contain multiple agents of both types within a socket or even combine the behavior of agents into a single education unit.)

[0041] In some implementations, the HPI can use an embedded clock. A clock signal can be embedded in data transmitted via the intermediary connection. With the clock signal embedded in the data, certain dedicated clock tracks can be omitted. This can be useful, for example, because it allows more pins of a device to be dedicated to data transfer, especially in systems where pin space is a priority.

[0042] The link layer can guarantee reliable data transfer between two protocol or routing units. The link layer can abstract the physical layer from the protocol layer, handle flow control between two protocol agents, and provide virtual channel services for the protocol layer (message classes) and routing layer (virtual networks).

[0043] In some implementations, the link layer can handle a fixed quantity of information called a flit. For example, the flit might be defined to have a length of 192 bits. However, any range of bits, such as 81–256 (or more), can be used in various combinations. A large flit size, such as 192 bits, can include formatting, cyclic redundancy checks (CRC), and other modifications. For instance, a larger flit length might also allow the CRC field to be expanded (e.g., to 16 bits) to handle the increased number of flit bits. The number of phits, or unit intervals (UI) (e.g., the time used to transfer a single bit or phit), used to transfer a single flit can vary with the link width.For example, among other potential examples, a 20-track or bit link width can transfer a single 192-bit flit in 9.6 UI, while an 8-track link width transfers the same flit in 24 UI. Link-layer crediting and protocol packaging can also be based on a flit.

[0044] Fig.Figure 6 illustrates a representation 600 of a generalized flit for an 8-lane connection width. Each column of the representation 600 can represent a connection lane, and each row a respective UI. In some implementations, a single flit can be divided into two or more slots. Specific messages or connection layer headers can be enclosed in each slot, allowing multiple specific and, in some cases, independent messages, potentially corresponding to different transactions, to be sent in a single flit. Furthermore, among other examples, the multiple messages enclosed in slots of a single flit can also be directed to different destination nodes. For example, the example of Fig. 6. A flit format with three slots. The shaded areas can represent the portion of the flit enclosed in each slot.

[0045] In the example of Fig. 6. An "Hdr" field can be provided for the flit in general and represent a header for the flit. In some cases, the Hdr field can indicate whether the flit is a header flit or a data flit. In data flits, the flit can still have slots, but the use of certain fields with payload data can be omitted or replaced. In some cases, data fields can include an operation code and payload data. In the case of header flits, a wide variety of header fields can be provided. In the example of Fig.6. “Oc” fields can be provided for each slot, where the Oc field represents an operation code. Similarly, one or more slots can have a corresponding “msg” field representing a message type of the corresponding packet to be included in the slot, provided the slot is configured to handle such packet types, etc. “DNID” fields can represent a destination node ID, a “TID” field can represent a transaction ID, and an “RHTID” field can represent either a requestor node ID or a home tracker ID, among other potential fields. Furthermore, one or more slots can be provided with payload fields. Additionally, among other examples, a CRC field can be included within a flit to provide a CRC value for the flit.

[0046] In some implementations, the link width can vary during the link's lifetime. For example, the physical layer can switch between link width states, such as to and from a full or initial track width and a different or partial track width. For instance, in some implementations, a link might be initialized to transfer data across 20 tracks. Later, among many other potential examples, the link might transition to a partial-width transfer state where only 8 tracks are actively used. Such track width transitions can be used, for example, in conjunction with power management tasks, which, among other examples, are managed by one or more power control units (PCUs).

[0047] As stated above, the connection width can affect the data throughput rate. Fig.Figure 7 is a representation of an exemplary 192-bit flit transmitted over an 8-lane connection, resulting in a flit throughput of 24 UI. Furthermore, as in the example of Fig. Figure 7 shows that in some cases, bits of the flit are sent out of order, for example, to send more time-sensitive fields earlier in the transfer (e.g., flit type fields (e.g., data or header flit), operation codes, etc.), and, among other examples, to maintain or simplify certain error detection or other functionality embodied in the flit. For example, in the example of Fig.7 bits 191, 167, 143, 119, 95, 71, 47, and 23 are sent in parallel on tracks L7 to L0 during the first UI (i.e., UI0) of the transfer, while bits 168, 144, 120, 96, 72, 48, 24, and 0 are sent during the 24th (or last) UI of the flit transfer (i.e., UI23). It is clear that other implementations and examples may use different sequencing schemes, flit lengths, track widths, etc.

[0048] In some cases, the length of the flit can be a multiple of the number of active tracks. In such cases, the flit can be sent evenly across all active tracks, and the transfer of the flit can essentially end simultaneously at a clear (i.e., non-overlapping) boundary. For example, as in the representation of Fig.Figure 8 shows that 8 bits of a flit are sent in consecutive groupings of 4 bits, or "tetrads." In this example, a 192-bit flit is to be sent over an 8-lane connection. Since 192 is a multiple of 8, the entire flit can clearly be transferred over the 8-lane connection in 24 UI. In other cases, the flit width cannot be a multiple of the number of active lanes. Fig.Figure 9, for example, shows another representation of an example 192-bit flit being transferred over 20 tracks. Since 192 is not evenly divisible by 20, transferring the entire flit would require a non-integer number of intervals (e.g., 9.6 UI). In such cases, instead of wasting "extra" tracks that are not used during the 10th UI of the transfer, a second overlapping flit can be transferred using the final bits of a previous flit. Such an overlap, or swizzling, of flits can lead to fuzzy flit boundaries and, in some implementations, to flit bits being sent out of order. The pattern used for the transfer can be designed to allow more time-sensitive fields of the flit to be transferred earlier in the flit, and, among other considerations, to maintain error detection and correction.Logic can be provided in one or both of the physical and interconnect layers to transfer flit bits according to such patterns and to dynamically switch between patterns based on the current interconnect width. Further logic can be provided, among other examples, to rearrange or reconstruct flits from such swizzling or ordered bit streams.

[0049] In some implementations, flits can be characterized as head flits (e.g., carrying packet header data) or data flits (e.g., carrying packet payload data). Referring again to Fig.6. A flit format can be defined that includes three (3) specific slots (e.g., 0, 1, and 2), allowing up to three heads to be transferred in a single flit (e.g., one head in each slot). Accordingly, each slot can contain both control fields and a payload information field. In addition to these, payload fields can be defined for each head (and slot). Furthermore, a floating payload information field can be defined, which can be flexibly used as additional payload length for two or more of the slots (e.g., either slot 0 or slot 1) based on the head types in those slots.

[0050] In one embodiment, such as the one associated with Fig.As illustrated in Figure 6, two slots, Slot 0 and Slot 1, can be defined to have equally sized payload fields, while Slot 2 has a much smaller payload field for use by a specific subset of heads that lack the use of such larger payload fields. Furthermore, in one example among other potential implementations, Slot 1 and 2 control fields can be configured to not carry full message class encodings (unlike Slot 0), and Slot 2 cannot carry full operation code encoding.

[0051] As stated above, in some implementations slots 1 and 2 cannot carry full message class encodings because not all bits are used due to slot education restrictions. Slot 1 can carry a message class bit 0. Here, request (REQ) and snoop (SNP) packets are allowed. In this implementation, REQ and SNP message class encodings are differentiated by a bit 0. If a developer wants to allow different message classes in a partial message class field, they can either choose a different bit position (i.e., an upper bit that differentiates two different types of messages) or assign different message types to the lower-order bit. Here, however, the upper two bits are implied as 0, with the lower bit distinguishing between REQ and SNP.In this example, slot 2 does not carry any message class bits because only response (RSP) (encoding 2) packets are allowed. Therefore, the message class encoding for slot 2 is RSP-2. Slot 2 can also carry a partial operation code. As above, one or more of the operation code bits can be assumed to be 0. As a result, partial message class fields and partial operation code fields can be used, defining a subset of messages and operation codes that can be used. Note that multiple sets of operation codes and message classes can be defined. If a lower-order message class bit is used, a subset of message types (i.e., MSG type 1 / MSG type 2) is available. However, if, as in other examples, 2 bits are used, then a larger subset is provided (e.g.,Message type 1 / Message type 2 / Message type 3 / Message type 4).

[0052] Message class encodings can correspond to specific header types that must be included in (or use) one or more defined slots in a flit. For example, the header can have multiple sizes. In one example, a three-slot flit can be defined to potentially support four header sizes based on the header type. Table 1 provides an example listing of potential header formats and associated sizes: TABLE 1 Head format Head size Description SA Single slot Requirement SA-S Single slot Snoops (includes floating useful information field) SA-D Single slot Data header SR-U small slot Completion without data SR-O Single slot arrange SR-C Single slot Conflict resolution SR-D Single slot Data header PW dual slot partial writing PR dual slot partial writing P2P dual slot Peer to peer NCM dual slot incoherent message transmission Slot-Zero Single slot (or operation code only) tax haven LLCRD small slot tax haven LLCTRL full flitt tax haven

[0053] Small (or single) slot headers can be used for messages small enough to fit in slot 2 and which have no protocol ordering requirements that force them into slot 0. A small slot header can also be placed in slot 0 if the slot-formation constraints for the flit require it. The single slot header can be used for messages with payload information that can fit in either slot 0 or slot 1. Some single slot headers can also use the floating payload field. For example, in one embodiment, Standard Address Snoop (SA-S) heads cannot be sent in both slot 0 and slot 1 of the same flit in the example where only one HTID or floating field exists. Certain single slot headers can use slot 0 based on protocol ordering requirements.The dual slot header can be, among other examples, those messages that are large enough to consume both the Slot 0 and Slot 1 payload fields in addition to the floating payload field.

[0054] A Slot NULL operation code can include special operation code that, in an example, can be used in either Slot 0 or Slot 1. For example, Slot_NULL can be used for Slot 0 if the link layer has no head to send in Slot 0 but does have a head to send in Slot 1 or 2. When Slot_NULL is used in Slot 0, it is assumed, among other examples, that the Slot 0 payload information is reserved (RSVD). In some implementations, Slot_NULL can potentially be used in Slot 1 under two conditions. First, if Slot 0 encodes a dual slot or special control head, thus consuming the Slot 1 payload information. In such cases, the Slot 1 operation code can be set to Slot_NULL. The second condition is if the link layer has nothing to send in Slot 1 but has a valid single-slot head for Slot 0 or a small-slot head for Slot 2.Under this condition, among other potential examples, the Slot 1 operation code can be set to Slot_NULL, and the Slot 1 payload information can be considered reserved.

[0055] In some implementations, the small Slot 2 may contain a reduced number of operation code bits. If the link layer has nothing to send in Slot 2, it can send "Implicit NULL" by encoding a specific operation code, such as a link layer credit operation code, and by setting the Slot 2 payload field to all zeros. The receiver of this Slot 2 encoding can process it as a link layer credit message (except in the case of special tax flits), but the all-zero encoding will have no effect on the credit and receipt state. In the case of special tax flits, since they can consume the entire flit, the Slot 2 payload can be viewed as RSVD, and implicit NULL is ignored. If, among other examples, the link layer has nothing to send in any of the three slots and CRD / ACK fields, the link layer can send a special tax-zero message.

[0056] Slot formation restrictions can be defined for one or more of the defined slots of a flit. In one embodiment, dual-slot headers can only have their message class and operation code placed in slot 0. If slot 0 contains a dual-slot header, slot 1 can encode a Slot_NULL operation code because the slot 1 payload field is consumed by the slot 0 header. If slot 0 has a Slot_NULL single slot or a small slot header, both slot 1 and slot 2 can encode a non-NULL head. In this particular example (e.g., in Fig. (As illustrated in Figure 6), only small slot heads are allowed in Slot 2. If both Slot 0 and Slot 1 contain single slot heads, one can be of a type that consumes the floating payload field. If neither Slot 0 nor Slot 1 has a head type that consumes the floating payload field, the field can be considered RSVD.

[0057] In some implementations, the link layer can additionally use several different types of virtual network or virtual channel loans. For example, pooled virtual network adaptive (VNA) loans can be supported, and a VNA field can be provided. In an example implementation, if the VNA field indicates a non-VNA flit (e.g., a flit that uses a different loan pool), the head can be specified to be placed in slot 0. Furthermore, the slot 2 operation code can include a Slot_2 loan in this case. If, among other potential implementations, slot 0 encodes a special control head, both slot 1 and slot 2 control fields can be set to fixed values, and no heads can be placed in these slots.

[0058] As stated above, various different fields can be provided in head flits to be enclosed in corresponding flits slots, as in the specific example of Fig. Figure 6 illustrates this. It should be noted that the illustrated and described fields are provided as an example and additional or supplementary fields may be included. In fact, in other examples, some of the described fields may be optional and omitted in some implementations.

[0059] In one example, a message class (MC) field and other fields may be provided. In some examples, the protocol layer may use the message class field to define the protocol class, which also acts as the primary operation code field. The link layer may use the message class field as part of the virtual channel (VC) definition. Some protocol classes / VCs, among other examples, may use multiple message class encodings due to the number of operation codes to be encoded. For example, requests (REQ), snoops (SNP), responses (RSP), writebacks, non-coherent bypasses, and non-coherent default types may be supported. If each type encodes sixteen operations, there is an operation code space of 96 operations.And if a different mode bit or operation code space is defined for each type, then another 96 operations can be provided, etc.

[0060] In one example, an operation code field can be provided. The protocol layer can use the operation code in conjunction with the message class to form a complete operation code (i.e., to define the message class type and the operation within it). For example, the same operation code with a REQ message type can define a first request operation, while the same operation code with an SNP message class can define a second, different SNP operation. The link layer can use the operation code to, for example, distinguish between a home agent and a cache agent destination for packets when a home agent and a cache agent share the same NodeID. Additionally, the link layer can also use the operation code, among other potential uses, to determine the packet size.

[0061] As stated above, flit heads can also include a virtual network-adaptive (VNA) field. In one example, setting the VNA field to a first value can indicate that the flit uses VNA credits. In other potential implementations, setting it to a second value indicates that the flit uses VN0 or VN1 credits. In one embodiment, a value can indicate that the flit is a single-slot flit, and the codes for slots 1 and 2 can be defined as NULL.

[0062] A virtual network (VN) field can also be provided to indicate to a flit whether the head(s) in the flit use a specific virtual network, such as VN0 or VN1. This can be used for both crediting purposes and to indicate which virtual network a message should clear if it uses VNA. If a VN bit is provided for the entire flit, any VNA flit containing multiple heads can ensure that all of them clear on VN0 or all of them clear on VN1. Alternatively, multiple VN bits can be provided. For non-VNA flits, it can be enabled that only slot 0 has a non-control operation code, allowing the VN to indicate this head's network.

[0063] In some implementations, slots in a flit can be used for small payload messages, such as loan returns, ACKs, and NAKs. In one example, a channel field can be provided that can be encoded for use with loan returns. This encoding, in combination with the virtual net field, can specify the virtual channel to which a loan return maps. If a message class has multiple encodings, they can all map to a single channel value for granting credit. If the loan return type is VNA, the channel value can be ignored. The use of RSVD encodings can be treated as an error by the receiving component. Table 2 includes examples of different channel options that can be encoded. Note that any combination of bits (or bits representing a hexadecimal value) can be used.As an example, a lower order of 3 bits can be used for encoding. TABLE 2 channel REQ: Request SNP: Snoop RSP: Response RSVD: Reserved WB: Write back NCB: non-coherent bypass NCS: non-coherent standard

[0064] Receipt or ACK fields can also be provided as header fields to be enclosed in a flits slot. An ACK field can be used by the link layer to communicate the successful reception of flits from a receiver to a sender. ACK with a first value indicates that a number of flits, such as 4, 8, or 12, have been received without errors. When a sender receives ACK, it can unassign the corresponding flits from the link layer retweet queue. ACK and ACK fields can be used in credit return control flits (e.g., LLCRD), where, among other examples, the total number of receipts returned is determined by the generation of the complete receipt return value (receipt part one, ACK, receipt part two).

[0065] As mentioned above, some implementations may also include a header indicator bit (Hdr), which can be used for one or more purposes. For example, an Hdr packet can identify whether the packet is a header or data flit, indicate that the flit is the start of a new packet, and indicate the start of a nested link-layer control flit. The Hdr can be set for the first flit of all packets.

[0066] As mentioned above, some implementations may provide special flit types, such as a link-layer control flit. Such control flits may continue to use the defined multi-slot format for a slot, while employing special header types corresponding to the control and error handling functionality that is simplified by using such control flits. For example, special header types may be provided that consume the entire flit and are used for communication between connected link layers.

[0067] In some implementations, special tax messages can be placed in a single message class plus operation code encoding for link-layer tax message transmission. This operation code can be called "LLCTRL," and all tax message types can fall under this subtype of operation code. This can, in some implementations, reduce the number of message class bits that need to be enclosed in a multi-slot flit format (e.g., from four message class bits to three, etc.). In some implementations, another form of link-layer tax flit can also be provided to handle a subset of tax messages. For example, among other examples, an LLCRD operation code can be defined for receipt and credit returns.In the case of LLCTRL operation codes, a multi-slot flit, instead of allowing the use of each of the multiple slots, can dedicate the entire flit payload information to the control message and allow special encodings for connection-to-connection communications.

[0068] Various control flits can be defined (e.g., under LLCTRL). For example, some implementations of an HPI intermediary might include transmitting a virus status in protocol-level messages and a poison status in data flits. In one embodiment, the protocol-level messages and the poison status can be moved to control flits. Since these bits are rarely used (e.g., only in the case of errors), removing them from the protocol-level messages potentially increases flit usage. Injecting them using control flits can further reduce errors.

[0069] Virus alerts can include a fault containment mechanism that arises from a fatal error, where preventing error propagation without immediately shutting down the system or suffering data corruption is difficult. A virus alert can address the fault propagation problem related to fatal errors, allowing an infected system to be shut down safely and, in the process, cleaning the system interface and other shared resources across system partitions.

[0070] Virus alerts can be implemented assuming the HPI interface is operational and can be used to provide the error indication. Within the HPI, a virus alert can be issued using a special virus error control flit. When an agent becomes viral, it will displace outgoing flits and send a viral flit to the remote agent. Any protocol agent that detects a fatal error or receives a packet indicating a viral state can enter a viral state. Once a viral state is established, the state can persist until the agent is reset (system reset) or some other platform-specific mechanism is used to clear the viral state. Once an agent becomes viral, all future packets from that agent are then assumed to be affected until the platform can determine the severity of the error.The platform may be responsible for controlling the system, so that masking the virus spread or cleaning up the viral state does not compromise fault containment. For example, I / O proxy units may stop routing arbitrary data to persistent storage or I / O devices after becoming viral. Additionally, one or more agents in a viral state may generate new requests to allow fault-handling software to shut down the system partition without problems. The mechanism used by the system for a trouble-free shutdown may be specific to the platform implementation and is beyond the scope of this description.

[0071] The virus alert mechanism can be a global state per partition and can be cleared during all reset events, including warm and cold resets. Under a virus alert, other outgoing flits are overridden by sending a virus error control flit. Fig. Figure 10 illustrates a representation of an example of a special virus error control flit 1005 on an 8-lane connection. As shown in this particular example, the general multi-slot format of a flit is maintained. However, in this example, message header fields of slot 0 are used to communicate the viral state. The remaining slots can be Slot_NULL as well as the payload information (e.g., interpreted as RSVD).

[0072] The link layer logic can, in some embodiments, be designed to restrict virus error control flits, causing them to be included in and enter a link layer retry queue. In fact, special control flits can be identified and handled differently from other flits, thus granting the flit priority. Furthermore, the structure of the special flit can be simplified, as in the example of Fig. 10, to make the processing of the tax flit more efficient. As one example among other features and examples, to ensure that a virus state is not lost in the event of an error in a virus error flit, a virus state can also be carried in an LLCTRL-RETRY-Ack message, for example.

[0073] In one embodiment, the link layer can additionally define three special diagnostic message types. A second set of standard diagnostic message types can be reserved for extensions to a future general diagnostic packet type. Support for diagnostic message types can be implementation- or device-specific. For example, LLCTRL-DEBUG flits can be sent by the link layer when an enable diagnostic control field is set. If this bit is not set, LLCTRL-DEBUG flits cannot be sent on the link. Diagnostic packets can be important for exposing internal states of devices connected via HPI that would otherwise be inaccessible. The contents of diagnostic packets can also be implementation-specific. The contents can include things like branching information (origin and destination IPs), timestamps, indications of an internal event trigger, and so on.The exposed data can originate, for example, from monitoring devices such as logic analyzers for post-processing and failure analysis. An example Flit that encodes a diagnostic message type is shown in [reference]. Fig. 11 illustrated.

[0074] HPI can further support the inclusion of poison fault communication using special flits. For example, a special poison fault control flit, such as the one (e.g., 1205) in the example of Fig.Figure 12 illustrates how poison can be used to inject poison into the data payload of a data packet, indicating that previously transmitted data has been determined to be corrupted or otherwise faulty. In some cases, a poison error control flit may apply to an immediately preceding flit on the same link. A special poison error control flit may be repeatable to guarantee that poison information is not lost in the event of a link failure. When data is to be poisoned, the link-layer poison error control flit is nested between the first and second data flits of a packet if the first 32 bytes need to be poisoned. If the second 32 bytes need to be poisoned, the poison error flit is nested between the second and third data flits, and so on.

[0075] In the particular example of Fig.In version 12, a poison error flit 1205 can encode the poison state in the operation code of slot 0, with the remaining slots (and corresponding fields) being coded with Slot_NULL. Furthermore, as in the example of the virus error control flit, user information fields can be set to zero or set to zero and considered RSVD fields.

[0076] Small slot fields can be used in some link-layer control messages. For example, credit (CRD) and acknowledgment (ACK) bits can be enclosed in small slots of a flit to allow, among many other examples, the return of a premapped number of credits, such as eight, or a number of ACKs, such as 8. In some implementations, credit and acknowledgment fields can be fully encoded to denote arbitrary credit or acknowledgment names. As an example, in fully encoded credit fields, bits can be used as Credit[n] and Acknowledge[n] when a slot is encoded to indicate that a link-layer credit (LLCRD) message is included.In some cases, full credit return fields can potentially improve efficiency by allowing any flit to return the number of loans and the number of receipts using a total of only 2 bits, but also by allowing their definitions to remain consistent when a full LLCRD return is used.

[0077] In one example, for flow control, credit / receipt information can flow as part of non-LLCTRL messages. For instance, in an implementation, HPI might provide that each header contains single-bit fields that serve as a mechanism for bulk credit repayments or bulk receipts. For example, setting such fields to "1" could indicate a repayment of 8 VNA credits (in the case of a CRD field) or 8 ACKs (in the case of an ACK field). This can allow credit repayments to be sent on any header (with the exception of LLCTRL messages in some implementations).

[0078] To address credit / receipt return increments other than any bulk or predefined set (e.g., 8) and to overcome the inefficiencies that can be introduced by a limited set of return increments, an LLCRD operation code can be provided. The LLCRD operation code can use the smallest slot in the header flit (e.g., slot 2) and encode to communicate credit and ACK returns in one or more formats. For example, in one example, a first format can be provided that allows a return of any set (e.g., from 0-7) of VN0 or VN1 credits for a single message class, and any set of ACKs (e.g., from 0-255) through bits dedicated to VN0 / 1 credit returns (e.g., 3 bits), other bits dedicated to an ACK return (e.g., 10 ...7 bits), and using the 'ACK' bit of the header bit as an acknowledge[2] to construct, for example, an 8-bit field. A second format can be provided that allows the return of any number (e.g., from 0-255) of VNA credits and any number of ACKs (e.g., from 0-255) through bits dedicated for VNA returns (e.g., 7 bits), and using the 'CRD' bit of the header flit as a credit[2] to construct, for example, an 8-bit field. Similarly, dedicated bits (e.g., 7 bits) can be provided for ACK returns, and the 'ACK' bit of the header flit can be used to construct, for example, an 8-bit field. These large, fully encoded fields can enable a sender to reimburse all credits or receipts that have been accumulated (e.g., buffered) in a single message.This can simplify the accumulated credit counting logic in some implementations to a simple "delete" instead of a decrementer on the accumulator.

[0079] In a specific example, which is in Fig. As illustrated in point 13, a flit can have a format like the one shown in the example of Fig. 6, as defined, can be used to support an LLCRD message. For example, in this particular example, an LLCRD message in slot 2 with a pre-trained operation code can be used to return VN0, VN1, and VNA credits, as well as ACKs for the link-layer retry queue. A link-layer credit (LLCRD) field (e.g., enclosed in "Value 1") can display the format for the LLCRD payload field (e.g., the examples below). Fig. 14 among other potential field formats).

[0080] Fig.Figure 13 shows a generic format for the slot in LLCRD message transmission. Fig.Figure 14 illustrates formats for two different LLCRD credit repayment messages, 1405 and 1410, that can be supported in Slot 2. For example, LLCRD format headers can be provided for both VN0 / 1 credit repayments (e.g., 1405) and VNA credit repayments (e.g., 1410). A Credit Repayment (CRD) field can be used to repay VNA credits across the connection. When set to a first value, this field indicates the repayment of a number of VNA credits, such as 4, 8, or 12. A Credit and Credit Repayment (CRDCRD) field can also be provided and can be used in LLCRD format headers for both VN credit repayments and VNA repayments. In a VN-LLCRD repayment format, the credit portion can display the total number of credits being repaid for the virtual network and message class.In a VNA-LLCRD repayment format, the total number of VNA loans to be repaid can be determined by generating the full VNA repayment value (e.g., part of the loan, the CRD, and a second part of the loan).

[0081] In a specific example, as in the examples of Fig.In a VN0 / 1-LLCRD return format (e.g., 1405), Credit[N:0] indicates the total number of returned credits for the virtual network and message class. In a VNA-LLCRD return format (e.g., 1410), the total number of VNA credits being returned is determined by generating the Full VNA[A:0] return value, where Full VNA[A:0] = {Credit[A:B], CRD, Credit[C:0]}. In some cases, a CRD field can also be used to refund VNA credits across the connection. When set to 1, this field indicates a refund of 8 VNA credits. If slot 2 encodes a VNA-LLCRD return type, the total number of returned VNA credits is as described below.

[0082] In some implementations of link-layer credit repayments, a channel field can be used to encode the channel for use in credit repayments. This encoding, in combination with the virtual network field, can be used to identify the virtual channel to which a credit repayment maps. If a message class has multiple encodings, they all map to a single channel value for crediting. If the credit repayment type is VNA, the channel value can be ignored.

[0083] As in the examples of Fig.As shown in Figure 14, ACK fields can also be included in the link layer message along with credit return fields. An ACK field can be used by the link layer to communicate the successful receipt of flits from a receiver to a sender. For example, ACK=1 can indicate that a certain number of flits were received without errors. When a sender receives an ACK, it can remove the corresponding flits from the link layer retry queue. In the examples of Fig. 14. Acknowledge [A:B] and Acknowledge [C:0] can be used to determine the total number of receipts to be returned by generating the Full_Acknowledge[A:0] return value, where Full_Acknowledge[A:0]={Acknowledge[A:B],ACK, Acknowledge[C:0]}.

[0084] In some implementations, certain fields can be defined to allow returns only in specific predefined increments. For example, in one example, among others, increments of 1 (for VN0 / 1), 2 / 8 / 16 (for VNA), and 8 (for Acknowledge) can be defined. This means that returning a large number of pending credits or receipts can use multiple return messages. It also means that odd-numbered return values ​​for VNA and Acknowledge can be left stranded before an accumulation of an integer-divisible value. Some HPI implementations can have fully coded credit and ACK return fields, allowing an agent to return all accumulated credits or ACKs for a pool with a single message.This potentially improves connection efficiency and also potentially simplifies logic implementation (feedback logic can implement a "reset" signal instead of a full decrementer).

[0085] In some implementations, credits for buffers in VN0 and VN1 can be reclaimed on a per-packet basis for each message class. Therefore, each buffer for each credit in VN0 / VN1 can be sized to meet the buffer requirements for the largest packet size that can use the credit. In some cases, this can provide the most efficient credit reclaim procedure for these channels.

[0086] Because of the shared resource and the wide variety of message sizes that can be allocated / released, using a packet credit / debit for VNA may not be efficient in some implementations. Instead, a flit credit / debit scheme for VNA can be used in some cases. Each flit can represent one flit of the receiver buffer space containing the credits shared by all message classes that can send to VNA. The encodings for credit repayment can be described in terms of the "LLCRD type." Furthermore, in some implementations, as mentioned above, a flit sent using VNA can contain up to three heads (one per slot). The receiver cannot repay a VNA credit until all three slots have been removed from the receiver queue, under other potential conditions or implementations.

[0087] HPI can be incorporated into any number of different computing devices and systems, including mainframes, server systems, personal computers, mobile computers (such as tablets, smartphones, personal digital systems, etc.), smart devices, gaming or entertainment consoles, and set-top boxes, among other examples. Fig. Figure 15 illustrates, for example, a sample computer system 1500 according to some implementations. As in Fig.As shown in Figure 15, a multiprocessor system 1500 is a point-to-point interconnect system comprising a first processor 1570 and a second processor 1580 coupled via a point-to-point interconnect 1550. Each of the processors 1570 and 1580 can be any version of a processor. In one embodiment, 1552 and 1554 are part of a serial coherent point-to-point interconnect fabric, such as a high-performance architecture. As a result, the invention can be implemented within the QPI architecture.

[0088] Although it is shown with only two processors 1570, 1580, it is clear that the scope of the present invention is not so limited. In other embodiments, one or more additional processors may be present in a given processor.

[0089] Processors 1570 and 1580 are shown to include integrated memory control units 1572 and 1582, respectively. Processor 1570 also includes point-to-point (PP) interfaces 1576 and 1578 as part of its bus control units; similarly, processor 1580 includes PP interfaces 1586 and 1588. Processors 1570 and 1580 can exchange information via a point-to-point (PP) interface 1550 using PP interface circuits 1578 and 1588. As shown in Fig. As shown in Figure 15, IMCs 1572 and 1582 couple the processors with respective memories, namely a memory 1532 and a memory 1534, which can be parts of a main memory, locally connected to the respective processors.

[0090] The 1570 and 1580 processors can each exchange information with a 1590 chipset via individual PP interfaces 1552 and 1554 using point-to-point interface circuits 1576, 1594, 1586, and 1598. The 1590 chipset also exchanges information with a high-performance graphics circuit 1538 via an interface circuit 1592 along a high-performance graphics intermediate link 1539.

[0091] A shared cache (not shown) can be enclosed within one of the two processors or outside of both processors, but connected to the processors via a PP intermediary, so that the local cache information of one or both processors can be stored in the shared cache if one processor is put into a power-saving mode.

[0092] The chipset 1590 can be coupled to a first bus 1516 via an interface 1596. In one embodiment, the first bus 1516 can be a Peripheral Component Interconnect (PCI) bus, a bus such as a PCI Express bus, or another third-generation I / O interconnect bus, although the scope of the present invention is not limited to this.

[0093] As in Fig.As shown in Figure 15, various I / O devices 1514 are coupled to the first bus 1516, together with a bus bridge 1518 that couples the first bus 1516 to a second bus 1520. In one embodiment, the second bus 1520 comprises a low pin count (LPC) bus. Various devices are coupled to the second bus 1520 in one embodiment, including, for example, a keyboard and / or mouse 1522, communication devices 1527, and a storage unit 1528 such as a disk drive or other mass storage device, which often includes instructions / code and data 1530. Furthermore, an audio I / O 1524 is shown coupled to the second bus 1520. It should be noted that other architectures are possible, with varying included components and interconnection architectures. For example, instead of the point-to-point architecture of Fig.15. Implement a system using a multidrop bus or other similar architecture.

[0094] Although the present invention has been described with respect to a limited number of embodiments, numerous modifications and variations thereof are apparent to those skilled in the art. The attached claims are intended to cover all such modifications and variations that fall within the basic concept and scope of the present invention.

[0095] Training can progress through various stages, from generation to simulation to fabrication. The data representing training can represent the training in several ways. First, as is useful in simulations, the hardware can be represented using a hardware description language or another functional description language. Additionally, a circuit-level model with logic and / or transistor gates can be generated at some stage of the training process. Furthermore, most trainings reach a level of data at some stage that represents the physical placement of various devices within the hardware model.In cases where conventional semiconductor manufacturing techniques are used, the data representing the hardware model can be data specifying the presence or absence of various features on different mask layers for masks used to create the integrated circuit. In one such representation, the data can be stored on any form of machine-readable medium. A memory or magnetic or optical storage medium, such as a disk, can serve as the machine-readable medium for storing information transmitted via optical or electrical waves, which are modulated or otherwise generated to transmit such information.When an electrical carrier wave is transmitted that displays or carries the code or training, a new copy is made to the extent that copying, buffering, or retransmission of the electrical signal is performed. In this way, a communications provider or network operator can store, at least temporarily, an item such as information encoded in a carrier wave on a tangible, machine-readable medium, embodying the techniques of embodiments of the present invention.

[0096] A module, as used here, refers to any combination of hardware, software, and / or firmware. For example, a module comprises hardware, such as a microcontroller, associated with a non-transitory medium for storing code adapted to be executed by the microcontroller. Therefore, in one embodiment, a reference to a module refers to the hardware specifically designed to recognize and / or execute the code to be held on a non-transitory medium. Furthermore, in another embodiment, the use of a module refers to the non-transitory medium that comprises the code specifically adapted to be executed by the microcontroller to perform predetermined operations.As can be deduced, in yet another embodiment the term module (in this example) can refer to the combination of the microcontroller and the non-transitory medium. Often, module boundaries, illustrated as separate, can typically vary and potentially overlap. For example, a first and a second module may share hardware, software, firmware, or combinations thereof, potentially retaining certain independent hardware, software, or firmware. In one embodiment, the use of the term logic includes hardware such as transistors, registers, or other hardware, such as programmable logic devices.

[0097] The use of the phrase "designed to" refers, in one embodiment, to arranging, assembling, manufacturing, offering for sale, introducing, and / or training a device, hardware, logic, or element to perform a designated or specific task. In this example, a device or element thereof that is not operating is still "designed" to perform a designated task if it is designated, coupled, and / or interconnected to perform the said designated task. As a purely illustrative example, a logic gate can output a 0 or a 1 during operation. However, a logic gate that is "designed" to provide a enable signal for a clock does not include every potential logic gate that can output a 1 or a 0. Instead, the logic gate is one that is coupled in some way so that, during operation, the output 1 or 0 enables the clock to be enabled.It should be noted again that the use of the term "designed" does not require operation, but instead focuses on the latent state of a device, hardware and / or element, wherein in the latent state the device, hardware and / or element is configured to perform a specific task when the device, hardware and / or element is in operation.

[0098] Furthermore, in one embodiment, the use of the terms "capable of" and "operable to" refers to any device, logic, hardware, and / or element configured to enable the use of the device, logic, hardware, and / or element in a specific manner. It should be noted, as above, that in one embodiment, the use of "capable of" and "operable to" refers to the latent state of a device, logic, hardware, and / or element, wherein the device, logic, hardware, and / or element is not operational but is configured to enable the use of a device in a specified manner.

[0099] A value, as used here, encompasses any known representation of a number, a state, a logic state, or a binary logic state. Often, the use of logic levels, logic values, or logical values ​​is also referred to as 1 and 0, which simply represent binary logic states. For example, 1 refers to a high logic level, and 0 refers to a low logic level. In one embodiment, a memory cell, such as a transistor or a flash cell, may be capable of holding a single logic value or multiple logic values. However, other representations of values ​​have been used in computer systems. For example, the decimal number ten can also be represented as the binary value 1010 and the hexadecimal letter A. Therefore, a value encompasses any representation of information that can be held in a computer system.

[0100] Furthermore, states can be represented by values ​​or parts of values. For example, a first value, such as a logical one, can represent a default or initial state, while a second value, such as a logical zero, can represent a non-default state. Additionally, in one embodiment, the terms reset and set refer to a default and an updated value or state, respectively. For instance, a default value potentially includes a high logic state, i.e., reset, while an updated value potentially includes a low logic state, i.e., set. It is important to note that any combination of values ​​can be used to represent any number of states.

[0101] The embodiments of methods, hardware, software, firmware, or code described above can be implemented via instructions or code stored on a machine-accessible, machine-readable, computer-accessible, or computer-readable medium that can be executed by a processing element. A non-transitory machine-accessible / readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form that can be read by a machine, such as a computer or electronic system.For example, a non-transitory machine-accessible medium includes random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM), ROM, magnetic or optical storage media, flash memory devices, electrical storage devices for holding information received from transitory (propagated) signals (e.g., carrier waves, infrared signals, digital signals), etc., which are to be distinguished from the non-transitory media which can receive information from them.

[0102] Instructions used to program logic for carrying out embodiments of the invention can be stored within a memory in the system, such as DRAM, cache, flash memory, or other storage. Furthermore, the instructions can be disseminated via a network or other computer-readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a form that can be interpreted by a machine (e.g., a computer).The term "computer-readable medium" encompasses any type of tangible, machine-readable medium capable of storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer). This includes, but is not limited to: floppy disks, optical discs, compact disc read-only storage (CD-ROMs) and magneto-optical disks, read-only storage (ROMs), random access storage (RAM), erasable programmable read-only storage (EPROM), electrically erasable programmable read-only storage (EEPROM), magnetic or optical cards, flash memory, or any tangible, machine-readable storage used in the transmission of information over the internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals). Accordingly, "computer-readable medium" includes any type of tangible, machine-readable medium suitable for storing or transmitting electronic instructions or information in a form that can be read by a machine (e.g., a computer).

[0103] The following examples relate to embodiments according to this description. One or more embodiments may provide a device, a system, a machine-readable memory, a machine-readable medium, and a method for generating a link-layer control message to be enclosed in a flit to be sent to a device via a serial data link, wherein flits sent via the data link must comprise a plurality of slots; and for sending the flit containing the link-layer control message.

[0104] In at least one example, the I / O logic comprises a layered stack that includes physical layer logic, link layer logic, and protocol layer logic.

[0105] In at least one example, the tax message is to be encoded in a specific one of the slots and no other messages are to be included in the slot.

[0106] In at least one example, the other slots must be coded as zero slots.

[0107] In at least one example, the tax message is at least partially identifiable from an encoding of an operation code of the specific flit.

[0108] In at least one example, the tax message includes a virus alert message.

[0109] In at least one example, the tax message includes a poison alert message.

[0110] In at least one example, the tax notification must include at least one loan repayment and receipt.

[0111] In at least one example, the tax message is to be encoded in a specific one of the plurality of slots, where the specific slot is the smallest of the plurality of slots.

[0112] In at least one example, the majority of slots consist of three defined slots.

[0113] In at least one example, the specific slot is the third of the three slots, the first of the three slots comprises 72 bits, the second of the three slots comprises 70 bits, and the third slot comprises 18 bits.

[0114] In at least one example, the tax message includes dedicated bits for virtual network loan repayments and dedicated bits for receipt repayments.

[0115] In at least one example, the dedicated bits for loan repayments comprise three bits adapted to be encoded with a loan repayment value between 0 and 7.

[0116] In at least one example, the dedicated bits for loan repayments comprise seven bits adapted to be encoded with a loan repayment value between 0 and 225.

[0117] In at least one example, the loan repayments include loan repayments to a shared adaptive virtual network (VNA) loan pool.

[0118] One or more embodiments may provide a device, a system, a machine-readable memory, a machine-readable medium and a method for receiving a flit over a serial data network, wherein flits have to comprise a plurality of slots, and for identifying a link layer control message from the flit.

[0119] In at least one example, the tax message is encoded in a specific one of the slots and no other messages are included in the slot.

[0120] In at least one example, the tax message includes a virus alert message.

[0121] In at least one example, the tax message includes a poison alert message.

[0122] In at least one example, the tax notification must include at least one of loan repayments and receipt returns.

[0123] In at least one example, the tax message is to be encoded in a specific one of the plurality of slots, where the specific slot is the smallest of the plurality of slots.

[0124] In at least one example, the tax message has a format that includes dedicated bits for virtual network loan returns and dedicated bits for receipt returns.

[0125] In at least one example, identifying the tax message involves identifying a set of accumulated receipts, and the tax message has to trace the receipts back.

[0126] In at least one example, identifying the tax message involves identifying a set of accumulated loans to be repaid, and the tax message has to repay the loans for a specific virtual network.

[0127] In at least one example, the control message can be communicated between a first and a second device. The first and second devices can be microprocessors, graphics accelerators, or other devices.

[0128] One or more aspects may further provide a layered protocol stack comprising at least one interconnection layer and a physical layer, wherein the layered protocol stack is designed to be coupled to a serial differential connection, and wherein the layered protocol stack is further designed to transmit a control fly comprising a poison state over the connection.

[0129] In at least one example, the control file includes link layer credit (LLCRD) message encoding.

[0130] In at least one example, the control file includes link layer control (LLCTRL) message encoding.

[0131] In at least one example, the tax field includes a fully coded loan repayment field.

[0132] One or more examples may further provide a layered protocol stack comprising at least one interconnect layer and a physical layer, wherein the layered protocol stack is designed to be coupled via a serial differential connection, and wherein the layered protocol stack is further designed to send a single flit comprising a combined credit and receipt to represent a first number of returned credits and a second number of receipts.

[0133] References to “an embodiment” throughout this entire description mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, references to the phrase “in an embodiment” at various points throughout this entire description do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0134] The above description provides a detailed account with reference to specific exemplary embodiments. However, it is clear that various modifications and changes can be made to it without deviating from the basic concept and scope of the invention, as set forth in the attached claims. Accordingly, the description and the drawings are to be considered illustrative rather than limiting. Furthermore, the above use of "embodiment" and other exemplary language does not necessarily refer to the same embodiment or example, but may refer to different and specific embodiments as well as potentially the same embodiment.

Claims

[1] Device comprising: I / O logic that is at least partially implemented in hardware to: to generate a link-layer control message to be enclosed in a flit to be sent to a device via a serial data link, wherein flits sent via the data link must comprise a plurality of slots, the slots enabling the encoding of a plurality of different headers in the flit, and wherein the control message is encoded in at least one of the slots in the flit; and to send the Flit with the link layer control message. [2] Device according to claim 1, wherein the I / O logic comprises a layered stack comprising physical layer logic, interconnect layer logic and protocol layer logic. [3] Device according to claim 1, wherein the control message is to be encoded in a specific of the slots and no other messages are to be included in the slot. [4] Device according to claim 3, wherein the other slots are to be coded as zero slots. [5] Device according to claim 3, wherein the control message is identifiable, at least partially, from an encoding of an operation code of the specified flit. [6] Device according to claim 3, wherein the control message includes a virus alert message. [7] Device according to claim 3, wherein the control message includes a poison alarm message. [8] Device according to claim 1, wherein the tax message includes at least one of loan repayments and receipts. [9] Device according to claim 8, wherein the control message is to be encoded in a specific slot of the plurality of slots, the specific slot being the smallest of the plurality of slots. [10] Device according to claim 9, wherein the plurality of slots consists of three defined slots. [11] Device according to claim 10, wherein the specified slot is the third of the three slots, the first of the three slots comprises 72 bits, the second of the three slots comprises 70 bits, and the third slot comprises 18 bits. [12] Device according to claim 8, wherein the control message comprises dedicated bits for virtual network loan returns and dedicated bits for receipt returns. [13] Device according to claim 12, wherein the dedicated bits for loan repayments comprise three bits adapted to be encoded with a loan repayment value between 0 and 7. [14] Device according to claim 12, wherein the dedicated bits for loan repayments comprise seven bits adapted to be encoded with a loan repayment value between 0 and 225. [15] Device according to claim 14, wherein the loan repayments comprise loan repayments to a shared adaptive virtual network (VNA) loan pool. [16] Device comprising: I / O logic that is at least partially implemented in hardware to: to receive a flit over a serial data network, wherein the flit comprises a plurality of slots according to a defined flit format; and a link layer control message consisting of data to be identified in a specific of the plurality of slots in the file, wherein the specific slot is the smallest of the plurality of slots and the link layer control message contains at least one credit return and one receipt return. [17] Device according to claim 16, wherein the control message has a format comprising dedicated bits for virtual network credit returns and dedicated bits for receipt returns. [18] Procedures, including: Identifying a link-layer control message to be sent to another device; Generating a flit to encompass the link-layer control message, wherein the flit has a defined format that defines a plurality of slots that allow encoding of a plurality of different headers in the flit, and wherein the control message is encoded in at least one of the slots in the flit and Sending the flit to the other device via a serial data link during a link transmission state. [19] System, encompassing: a first device; and a second device that is communicatively coupled to the first device using a serial data link, wherein the second device comprises a link layer module executed by at least one processor to: to identify a link layer control message to be sent to the first device; to generate a flit to encompass the link-layer control message, wherein the flit has to encompass a plurality of slots and the control message is encoded in at least one of the plurality of slots in the flit; and to send the Flit to the other device via a serial data link during a link transmission state. [20] System according to claim 19, wherein the first device comprises a microprocessor. [21] System according to claim 20, wherein the second device comprises a second microprocessor. [22] System according to claim 20, wherein the second device comprises a graphics accelerator. [23] System according to claim 19, wherein the first device comprises interconnect layer logic to: to receive the Flit via the data connection; to identify the tax message from the Flit; and to process the tax message.

Images