Shared Buffer Memory Routing

Abstract

A shared memory controller receives a microchip from another first shared memory controller via a shared memory link, wherein the microchip includes a node identifier (ID) field and an address of a specific row of a shared memory. The node ID field identifies the source of the first shared memory controller corresponding to the microchip. In addition, a second shared memory controller is determined based on at least the address field of the microchip, wherein the second shared memory controller is connected to the memory element corresponding to the specific row. The microchip is forwarded to the second shared memory controller using the shared memory link according to the routing path.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Non-Provisional Patent Application No. 14 / 670,578, filed on March 27, 2015, entitled “SHARE BUFFERED MEMORY ROUTING,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to computing systems and, in particular, but not exclusively, to memory access between components in a computing system. Background Art

[0004] Advances in semiconductor processing and logic design have allowed for an increase in the amount of logic that can be present on an integrated circuit device. As a corollary, computer system configurations have evolved from a single or multiple integrated circuits in a system to multiple cores, multiple hardware threads, and multiple logical processors present on each integrated circuit and other interfaces integrated within such processors. A processor or integrated circuit typically includes a single physical processor die, where the processor die may include any number of cores, hardware threads, logical processors, interfaces, memory, controller hubs, etc.

[0005] Smaller computing devices are becoming more popular due to the greater ability to fit more processing power in a smaller package. Smart phones, tablets, ultra-thin notebooks, and other user devices have grown exponentially. However, these smaller devices rely on servers for both complex processing and data storage that exceed the form factor. Therefore, the demand in the high-performance computing market (i.e., server space) has also increased. For example, in modern servers, there is usually not only a single processor with multiple cores, but also multiple physical processors (also known as multiple sockets) to increase computing power. But as processing power grows along with the number of devices in a computing system, communication between sockets and other devices becomes more critical.

[0006] In fact, interconnects have grown from more traditional multi-drop buses that originally handled electrical communications to fully developed interconnect architectures that facilitate fast communications. Unfortunately, with the need for future processors to consume at even higher rates, corresponding demands are placed on the capabilities of existing interconnect architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An embodiment of a computing system including an interconnect architecture is illustrated.

[0008] Figure 2 An embodiment of an interconnect architecture including a layered stack is illustrated.

[0009] Figure 3 The diagram illustrates embodiments of requests or packets to be generated or received within an interconnect fabric.

[0010] Figure 4 An embodiment of a transmitter and receiver pair for an interconnect architecture is shown.

[0011] Figure 5 An embodiment of a layered protocol stack associated with a high-performance general purpose input / output (GPIO) interconnect is illustrated.

[0012] Figure 6 A representation of an example multi-grooved microchip is shown.

[0013] Figure 7 An example system utilizing buffered memory access is illustrated.

[0014] Fig. 8A A simplified block diagram of an embodiment of an example node is shown.

[0015] Figure 8B A simplified block diagram of an embodiment of an example system comprising multiple nodes is shown.

[0016] Figure 8C Another simplified block diagram of an embodiment of an example system comprising multiple nodes is shown.

[0017] Fig. 9 is a representation of data transferred according to an example shared memory link.

[0018] Fig. 10A is a representation of data transferred according to another example of a shared memory link.

[0019] Fig. 10B is a representation of the start of an example of a data framing token.

[0020] Fig.11 is a representation of data transferred according to another example of a shared memory link.

[0021] Fig.12 A representation of an example multi-slot flit modified for routing within a shared memory architecture is shown.

[0022] Figures 13A-13B is a flow diagram illustrating an example technique for routing transactions within a shared memory architecture.

[0023] Fig.14 An embodiment of a block diagram for a computing system including a multi-core processor is illustrated.

[0024] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0025] In the following description, numerous specific details are set forth, such as the following examples: specific processor types and system configurations, specific hardware structures, specific architecture and microarchitecture details, specific register configurations, specific instruction types, specific system components, specific measurements / heights, specific processor pipeline stages and operations, etc., in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that these specific details need not be employed to practice the present invention. In other instances, well-known components or methods (such as specific and alternative processor architectures, specific logic circuits / codes for the described algorithms, specific firmware codes, specific interconnect operations, specific logic configurations, specific manufacturing techniques and materials, specific compiler implementations, specific expressions of algorithms in code, specific power-down and gating techniques / logic, and other specific operational details of computer systems) are not described in detail in order to avoid unnecessarily obscuring the present invention.

[0026] Although the following embodiments may be described with reference to energy conservation and energy efficiency in a specific integrated circuit (such as in a computing platform or microprocessor), other embodiments may be applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of the embodiments described herein may be applied to other types of circuits or semiconductor devices that may also benefit from better energy efficiency and energy conservation. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks. TM (Ultrabook). Rather, it can also be used in other devices such as handheld devices, tablets, other thin notebooks, system-on-chip (SOC) devices, and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically include microcontrollers, digital signal processors (DSPs), system-on-chips, network computers (NetPCs), set-top boxes, network hubs, wide area network (WAN) switches, or any other system that can perform the functions and operations taught below. In addition, the devices, methods, and systems described herein are not limited to physical computing devices, but may also involve software optimization for energy conservation and energy efficiency. As will become readily apparent in the following description, embodiments of the methods, devices, and systems described herein (whether with reference to hardware, firmware, software, or a combination thereof) are critical to the future of "green technology" that is weighed against performance considerations.

[0027] As computing systems continue to advance, the components therein are becoming more complex. As a result, the interconnect architectures that couple components and communicate between components are also increasing in complexity to ensure that bandwidth requirements are met for optimal component operation. In addition, different market segments require different aspects of the interconnect architecture to adapt to market demands. For example, servers require higher performance, while mobile ecosystems can sometimes sacrifice overall performance to save power. However, the sole purpose of most configurations is to provide the highest possible performance with maximum power savings. A variety of interconnects are discussed below that will potentially benefit from various aspects of the invention described herein.

[0028] One interconnect fabric architecture includes the Peripheral Component Interconnect Express (PCI) (PCIe) architecture. The primary goal of PCIe is to enable components and devices from different vendors to interoperate in an open architecture, across multiple market segments; clients (desktop and mobile), servers (standard and enterprise), and embedded and communication devices. PCI Express is a high-performance, general-purpose I / O interconnect that is defined for a wide variety of future computing and communication platforms. Some PCI attributes, such as its usage model, load-store architecture, and software interface, have been maintained through its revisions, while previous parallel bus implementations have been replaced by highly scalable, all-serial interfaces. Recent versions of PCI Express leverage advances in point-to-point interconnects, switch-based technologies, and packetized protocols to deliver new levels of performance and features. Power management, quality of service (QoS), hot-plug / hot-swap support, data integrity, and error handling are among some of the advanced features supported by PCI Express.

[0029] refer to Figure 1 , illustrates an embodiment of a construction consisting of a point-to-point link interconnecting a set of components. System 100 includes a processor 105 and a system memory 110 coupled to a controller hub 115. Processor 105 includes any processing element, such as a microprocessor, a main processor, an embedded processor, a coprocessor, or other processor. Processor 105 is coupled to controller hub 115 via a front side bus (FSB) 106. In one embodiment, FSB 106 is a serial point-to-point interconnect as described below. In another embodiment, link 106 includes a serial, differential interconnect architecture that complies with different interconnect standards.

[0030] System memory 110 includes any memory device, such as random access memory (RAM), non-volatile (NV) memory, or other memory accessible by devices in system 100. System memory 110 is coupled to controller hub 115 via memory interface 116. Examples of memory interfaces include double data rate (DDR) memory interfaces, dual channel DDR memory interfaces, and dynamic RAM (DRAM) memory interfaces.

[0031] In one embodiment, controller hub 115 is a root hub, root complex, or root controller in the Peripheral Component Interconnect Express (PCIe or PCIE) interconnect hierarchy. Examples of controller hub 115 include chipsets, memory controller hubs (MCHs), north bridges, interconnect controller hubs (ICHs), south bridges, and root controllers / hubs. The term chipset often refers to two physically separate controller hubs, namely, a memory controller hub (MCH) coupled to an interconnect controller hub (ICH). Note that current systems often include an MCH integrated with processor 105, and controller 115 will communicate with I / O devices in a similar manner as described below. In some embodiments, peer-to-peer routing is optionally supported through root complex 115.

[0032] Here, controller hub 115 is coupled to switch / bridge 120 via serial link 119. Input / output modules 117 and 121, which may also be referred to as interfaces / ports 117 and 121, include / implement a layered protocol stack to provide communication between controller hub 115 and switch 120. In one embodiment, multiple devices can be coupled to switch 120.

[0033] The switch / bridge 120 routes packets / messages from the device 125 upstream (i.e., above the hierarchy) toward the root complex to the controller hub 115 and downstream (i.e., below the hierarchy away from the root controller), from the processor 105 or the system memory 110 to the device 125. In one embodiment, the switch 120 is referred to as a logical assembly of multiple virtual PCI to PCI bridge devices. The device 125 includes any internal or external device or component to be coupled to the electronic system, such as an I / O device, a network interface controller (NIC), an expansion card, an audio processor, a network processor, a hard drive, a storage device, a CD / DVD ROM, a monitor, a printer, a mouse, a keyboard, a router, a portable storage device, a FireWire device, a Universal Serial Bus (USB) device, a scanner, and other input / output devices. Usually in PCIe lingo, such as a device, is referred to as an endpoint. Although not specifically shown, the device 125 may include a PCIe to PCI / PCI-X bridge to support legacy or other versions of PCI devices. Endpoint devices in PCIe are often classified as legacy endpoints, PCIe endpoints, or endpoints with integrated root complexes.

[0034] The graphics accelerator 130 is also coupled to the controller hub 115 via a serial link 132. In one embodiment, the graphics accelerator 130 is coupled to the MCH, which is coupled to the ICH. The switch 120, and accordingly the I / O devices 125, are then coupled to the ICH. The I / O modules 131 and 118 will also implement a layered protocol stack to communicate between the graphics accelerator 130 and the controller hub 115. Similar to the MCH discussion above, the graphics controller or graphics accelerator 130 itself can be integrated into the processor 105.

[0035] Go to Figure 2 , illustrates an embodiment of a layered protocol stack. The layered protocol stack 200 includes any form of layered communication stack, such as a Quick Path Interconnect (QPI) stack, a PCIe stack, a next generation high performance computing interconnect stack, or other layered stack. Although the reference immediately below Figure 1-4 The discussion is related to the PCIe stack, but the same concepts can be applied to other interconnect stacks. In one embodiment, the protocol stack 200 is a PCIe protocol stack, which includes a transaction layer 205, a link layer 210, and a physical layer 220. Figure 1 Interfaces such as interfaces 117, 118, 121, 122, 126, and 131 in may be represented as a communication protocol stack 200. A representation as a communication protocol stack may also be referred to as a module or interface that implements / includes the protocol stack.

[0036] PCI Express uses packets to transfer information between components. Packets may be formed in the transaction layer 205 and the data link layer 210 to convey information from a transmitting component to a receiving component. As the transmitted packets flow through the other layers, they are expanded with additional information necessary to process the packets at those layers. The reverse process occurs on the receiving side and the packet is transformed from its physical layer 220 representation to a data link layer 210 representation and finally (for a transaction layer packet) to a form that can be processed by the transaction layer 205 of the receiving device.

[0037] Transaction Layer

[0038] In one embodiment, transaction layer 205 is used to provide an interface between a device's processing core and an interconnect architecture (such as data link layer 210 and physical layer 220). In this regard, the primary responsibility of transaction layer 205 is the assembly and disassembly of packets (i.e., transaction layer packets or TLPs). Transaction layer 205 typically manages credit-based flow control for TLPs. PCIe implements split transactions, i.e., transactions with requests and responses divided in time, allowing the link to carry other traffic while the target device collects data for the response.

[0039] In addition, PCIe utilizes credit-based flow control. In this scenario, the device advertises an initial amount of credits for each receive buffer in the transaction layer 205. An external device at the opposite end of the link, such as Figure 1 The controller hub 115 in the , counts the number of credits consumed by each TLP. If the transaction does not exceed the credit limit, the transaction can be transmitted. When a response is received, the credit amount is restored. The advantage of the credit scheme is that if the credit limit is not counted, the waiting time for the credit to be returned does not affect performance.

[0040] In one embodiment, the four transaction address spaces include a configuration address space, a memory address space, an input / output address space, and a message address space. The memory space transaction includes one or more of a read request and a write request to transfer data to or from a memory mapped location. In one embodiment, the memory space transaction can use two different address formats, for example, a short address format (such as a 32-bit address) or a long address format (such as a 64-bit address). The configuration space transaction is used to access the configuration space of the PCIe device. Transactions for the configuration space include a read request and a write request. Message space transactions (or simply, messages) are defined to support in-band communication between PCIe agents.

[0041] Thus, in one embodiment, transaction layer 205 assembles packet header / payload 206. The current format of packet header / payload can be found in the PCIe specification at the PCIe specification website.

[0042] Quick Reference Figure 3 , illustrates an embodiment of a PCIe transaction descriptor. In one embodiment, a transaction descriptor 300 is a mechanism for carrying transaction information. In this regard, the transaction descriptor 300 supports the identification of transactions in the system. Other potential uses include tracking modifications to default transaction ordering and association of transactions with channels.

[0043] Transaction descriptor 300 includes a global identifier field 302, an attribute field 304, and a channel identifier field 306. In the illustrated example, global identifier field 302 is depicted as including a local transaction identifier field 308 and a source identifier field 310. In one embodiment, global transaction identifier 302 is unique to all outstanding requests.

[0044] According to one implementation, the local transaction identifier field 308 is a field generated by the requesting agent and is unique to all outstanding requests requiring completion for the requesting agent. Additionally, in this example, the source identifier 310 uniquely identifies the requesting agent within the PCIe hierarchy. Thus, along with the source ID 310, the local transaction identifier 308 field provides a global identification of a transaction within the hierarchy domain.

[0045] Attribute field 304 specifies the characteristics and relationships of the transaction. In this regard, attribute field 304 is potentially used to provide additional information that allows modification of the default processing of the transaction. In one embodiment, attribute field 304 includes priority field 312, reserved field 314, sort field 316, and no-snoop field 318. Here, priority subfield 312 can be modified by the initiator to assign priorities to transactions. Reserved attribute field 314 is reserved for future use or vendor-defined use. Possible use models using priority or security attributes can be implemented using reserved attribute fields.

[0046] In this example, the sort attribute field 316 is used to provide optional information that conveys the type of sorting that can modify the default sorting rules. According to an example implementation, a sort attribute of "0" indicates that the default sorting rules are applied, wherein a sort attribute of "1" indicates a relaxed sorting, wherein writes can pass writes in the same direction, and read completions can pass writes in the same direction. The listening attribute field 318 is used to determine whether the transaction is listened. As shown, the channel ID field 306 identifies the channel with which the transaction is associated.

[0047] Link Layer

[0048] The link layer 210 (also referred to as the data link layer 210) acts as an intermediate level between the transaction layer 205 and the physical layer 220. In one embodiment, the responsibility of the data link layer 210 is to provide a reliable mechanism for exchanging transaction layer packets (TLPs) between two components on a link. One side of the data link layer 210 accepts TLPs assembled by the transaction layer 205, applies a packet sequence identifier 211 (i.e., an identification number or packet number), calculates and applies an error detection code (i.e., CRC 212), and presents the modified TLP to the physical layer 220 for transmission across the physical to an external device.

[0049] Physical Layer

[0050] In one embodiment, the physical layer 220 includes a logical sub-block 221 and an electrical sub-block 222 to physically transmit packets to an external device. Here, the logical sub-block 221 is responsible for the "digital" functions of the physical layer 221. In this regard, the logical sub-block includes a transmission section to prepare outgoing information for transmission by the physical sub-block 222, and a receiver section to identify and prepare received information before delivering it to the link layer 210.

[0051] The physical electrical sub-block 222 includes a transmitter and a receiver. The transmitter is provided by the logic sub-block 221 using symbols, and the transmitter serializes the symbols and transmits them to an external device. The receiver is provided with serialized symbols from the external device and converts the received signal into a bit stream. The bit stream is deserialized and provided to the logic sub-block 221. In one embodiment, an 8b / 10b transmission code is used, in which 10-bit symbols are transmitted / received. Here, special symbols are used to construct packets using frames 223. In addition, in one example, the receiver also provides a symbol clock recovered from the incoming serial stream.

[0052] As described above, although the transaction layer 205, link layer 210, and physical layer 220 are discussed with reference to a specific embodiment of the PCIe protocol stack, the layered protocol stack is not limited thereto. In fact, any layered protocol may be included / implemented. As an example, a port / interface represented as a layered protocol includes: (1) a first layer, i.e., a transaction layer, for assembling packets; a second layer, i.e., a link layer, for sorting packets; and a third layer, i.e., a physical layer, for transmitting packets. As a specific example, a common standard interface (CSI) layered protocol is utilized.

[0053] Next reference Figure 4 , an embodiment of a PCIe serial point-to-point structure is illustrated. Although an embodiment of a PCIe serial point-to-point link is illustrated, the serial point-to-point link is not limited thereto because it includes any transmission path for transmitting serial data. In the illustrated embodiment, the basic PCIe link includes two low-voltage differentially driven signal pairs: a transmission pair 406 / 411 and a reception pair 412 / 407. Thus, the device 405 includes a transmission logic 406 for transmitting data to the device 410, and a reception logic 407 for receiving data from the device 410. In other words, two transmission paths, namely paths 416 and 417, and two reception paths, namely paths 418 and 419, are included in the PCIe link.

[0054] A transmission path refers to any path for transmitting data, such as a transmission line, copper wire, optical line, wireless communication channel, infrared communication link, or other communication path. A connection between two devices, such as the connection between device 405 and device 410, is referred to as a link, such as link 415. A link may support one channel - each channel represents a set of differential signal pairs (one pair for transmission and one pair for reception). To scale bandwidth, a link may aggregate multiple channels, represented as xN, where N is any supported link width, such as 1, 2, 4, 8, 12, 16, 20, 24, 32, 64, or wider.

[0055] A differential pair refers to two transmission paths, such as lines 416 and 417, to transmit differential signals. As an example, when line 416 is toggled from a low voltage level to a high voltage level (i.e., a rising edge), line 417 is driven from a high logic level to a low logic level (i.e., a falling edge). Differential signals potentially exhibit better electrical characteristics, such as better signal integrity (i.e., cross-coupling), voltage overshoot / undershoot, ringing, etc. This allows for better timing windows, which can enable faster transmission frequencies.

[0056] In one implementation, if Figure 5 As shown in , physical layer 505a, b can be responsible for quickly transmitting information on physical media (electrical or optical, etc.). A physical link can be point-to-point between two link layer entities (such as layers 505a and 505b). Link layer 510a, b can extract physical layer 505a, b from the upper layer and provide the ability to reliably transmit data (and requests) and manage flow control between two directly connected entities. The link layer can also be responsible for virtualizing the physical channel into multiple virtual channels and message classes. Protocol layer 520a, b relies on link layer 510a, b to map protocol messages to appropriate message classes and virtual channels before processing protocol messages to physical layer 505a, b for transmission across physical links. Link layer 510a, b can support multiple messages, such as requests, monitoring, responses, write-backs, non-consistent data, and other examples.

[0057] The physical layer 505a, b (or PHY) may be implemented above the electrical layer (ie, the electrical conductors connecting two components) and below the link layer 510a, b, such as Figure 5As shown in the figure. The physical layer and corresponding logic may reside on each agent and connect the link layer on two separate agents or nodes (A and B) (for example, on the devices on either side of the link). The local electrical layer and the remote electrical layer are connected by a physical medium (for example, wires, conductors, optics, etc.). In one embodiment, the physical layer 505a, b has two main phases, initialization and operation. During initialization, the connection is opaque to the link layer, and signaling may involve a combination of timing states and handshake events. During operation, the connection is transparent to the link layer and signaling is at a certain speed, where all channels operate together as a single link. During the operation phase, the physical layer transmits microchips 535 from agent A to agent B and from agent B to agent A. The connection is also called a link, and some physical aspects are extracted from the link layer, including media, width and speed, while exchanging microchips and control / states of the current configuration (for example, width) with the link layer. The initialization phase includes secondary phases, such as polling and configuration. The operation phase also includes secondary phases (for example, link power management state).

[0058] In one embodiment, the link layer 510a, b may be implemented to provide reliable data transmission between two protocol or routing entities. The link layer may extract the physical layer 505a, b from the protocol layer 520a, b, and may be responsible for flow control between two protocol agents (A, B), and provide virtual channel services to the protocol layer (message class) and the routing layer (virtual network). The interface between the protocol layer 520a, b and the link layer 510a, b may generally be at the packet level. In one embodiment, the minimum transmission unit at the link layer is called a flit, which is a specified number of bits, such as 192 bits or some other amount. The link layer 510a, b relies on the physical layer 505a, b to construct the 505a, b transmission unit (physical flit 540) of the physical layer into the 510a, b transmission unit (flit) of the link layer. In addition, the link layer 510a, b may be logically divided into two parts, namely a transmitter and a receiver. A transmitter / receiver pair on one entity may be connected to a receiver / transmitter pair on another entity. Flow control is typically performed on both a flit and packet basis. Error detection and correction is also potentially performed on a flit-level basis.

[0059] In one embodiment, the routing layer 515a, b can provide a flexible and distributed method to route transactions from a source to a destination. The scheme is flexible because the routing algorithms for multiple topologies can be specified by a programmable routing table at each router (in one embodiment, the programming is performed by firmware, software, or a combination thereof). The routing functionality can be distributed; routing can be completed by a series of routing steps, each of which is defined by a lookup of a table at any of the source, intermediate, or destination routers. The lookup at the source can be used to inject packets into the structure. The lookup at the intermediate router can be used to route packets from the input port to the output port. The lookup at the destination port can be used to target the destination protocol agent. Note that in some implementations, the routing layer may be thinner because the routing table and therefore the routing algorithm are not specifically defined by the specification. This allows flexibility and a variety of usage models, including a flexible platform architecture topology to be defined by the system implementation. The routing layer 515a,b relies on the link layer 510a,b to provide the use of up to three (or more) virtual networks (VNs) - in one example, two deadlock-free VNs, VN0 and VN1, with several message classes defined in each virtual network. A shared adaptive virtual network (VNA) may be defined in the link layer, but the adaptive network may not be directly exposed in the routing concept because each message class and virtual network may have dedicated resources and guaranteed forward progress, as well as other features and examples.

[0060] In one embodiment, protocol layers 520a,b may provide a consistency protocol to support agents that cache data lines from memory. Agents that wish to cache memory data may use the consistency protocol to read data lines to load into their cache. Agents that wish to modify data lines in their cache may use the consistency protocol to obtain ownership of the line before modifying the data. After modifying the line, the agent may follow the following protocol requirements: keep it in its cache until the line is written back to memory or the line is included in the response to an external request. Finally, the agent may satisfy the external request to invalidate the line in its cache. The protocol ensures data consistency by specifying rules that all caching agents may follow. It also provides a means for agents without cache to read and write memory data consistently.

[0061] The physical layer of existing interconnect and communication architectures (including PCIe) can be utilized to provide shared memory and I / O services within the system. Traditionally, cacheable memory cannot be shared between independent systems using traditional load / store (LD / ST) memory semantics. An independent system or "node" can be independent in the sense that it acts as a single logical entity, is controlled by a single operating system (and / or a single BIOS or virtual machine monitor (VMM)), and / or has independent fault domains. A single node may include one or more processor devices, may be implemented on a single board or multiple boards, and may include local memory, including cacheable memory that can be accessed by devices on the same node using LD / ST semantics. Within a node, shared memory may include one or more memory blocks (such as random access memory (RAM)) that can be accessed by several different processors (e.g., central processing units (CPUs)) within the node. Shared memory may also include local memory of processors or other devices in a node. Multiple devices within a node with shared memory can share a single view of data within the shared memory. I / O communications involving shared memory can have very low latency and allow fast access to memory by multiple processors.

[0062] Traditionally, memory shared between different nodes does not allow memory sharing according to the load / store paradigm. For example, in some systems, memory sharing between different nodes has been promoted by a distributed memory architecture. In traditional solutions, computing tasks operate on local data, and if data from another node is expected, the computing task (e.g., performed by another CPU node) communicates with the other node (e.g., by utilizing a communication channel of a communication protocol stack such as Ethernet, Infiniband or another layered protocol). In traditional multi-node systems, processors of different nodes do not have to know where the data resides. Sharing data using traditional methods (such as through a protocol stack) can have significantly higher latency than memory sharing within a node using a load / store paradigm. In contrast to directly addressing and operating on data in shared memory, a node can request data from another node using an existing protocol handshake such as Ethernet (or Infiniband), and the source node can provide data so that the data can be stored and operated on by the requesting node, as well as other examples.

[0063] In some implementations, a shared memory architecture may be provided that allows memory to be shared between independent nodes for exclusive or shared access using load / store (LD / ST) memory semantics. In one example, memory semantics (and directory information, if applicable) as well as I / O semantics (for protocols such as PCIe) may be derived on a common pin group or a separate pin group. In such a system, an improved shared memory architecture may enable each of multiple nodes in the system to maintain its own independent fault domain (and local memory) while enabling a shared memory pool to be accessed by the nodes and enabling low latency messages to be passed between nodes using memory according to LD / ST semantics. In some implementations, such a shared memory pool may be dynamically (or statically) allocated between different nodes. Thus, the various nodes of the system may also be configured into dynamically changing groups of nodes to work collaboratively and flexibly on various tasks utilizing a shared memory infrastructure, for example, when the need arises.

[0064] In some implementations, the shared memory architecture can be based on a buffer memory interface. The buffer memory interface itself can be based on a general purpose input / output (GPIO) interconnect interface and protocol. For example, the physical layer and link layer definitions of the GPIO interconnect can also be implemented in the buffer memory protocol. In fact, the logic used to support the physical layer and link layer of the GPIO protocol can be reused at the interface supporting the buffer memory protocol. The buffer memory protocol can also share message classes, such as request, response, and write-back message classes, as well as other examples. Although the opcode values ​​within the buffer memory protocol messages can be interpreted differently than in the GPIO protocol, the same general packet and micro-chip format can be utilized in both the buffer memory protocol and the GPIO interconnect on which it is built.

[0065] In one example, a flits format may be defined for flits sent between agents in a GPIO protocol. Figure 6 A representation 600 of a generalized microslice for an 8-lane link width is illustrated. Each column of representation 600 can symbolize a link lane, and each row can symbolize a corresponding UI. In some implementations, a single microslice can be subdivided into two or more slots. Each slot can contain a different message or link layer header to allow multiple different (and in some cases independent) messages corresponding to potentially different transactions to be sent in a single microslice. In addition, multiple messages included in a slot of a single microslice can also go to different destination nodes, among other examples. For example, Figure 6 The example diagram of FIG. 1 shows a microsheet format having three slots. The shaded portion may represent the portion of the microsheet included in the corresponding slot.

[0066] exist Figure 6In the example of , three slots are provided, slots 0, 1 and 2. Slot 0 may be provided with 72 bits of microchip space, of which 22 bits are dedicated to message header fields and 50 bits are used for message payload space. Slot 1 may be provided with 70 bits of microchip space, of which 20 bits are dedicated to message header fields and 50 bits are used for message payload space. The difference between the message header field spaces may be optimized to provide that certain message types will be designated as included in slot 0 (e.g., where more message header encoding is utilized). A third slot (slot 2) may be provided that occupies much less space than slots 0 and 1, in which case 18 bits of microchip space are utilized. Slot 2 may be optimized to handle messages that do not employ larger message payloads, such as acknowledgments, credit returns, etc. In addition, a floating payload field may be provided that allows an additional 11 bits to be applied alternatively to supplement the payload field of slot 0 or slot 1.

[0067] continue Figure 6 For example, other fields may be global to the flits, (i.e., apply across flits rather than to a specific slot). For example, header bits may be provided along with a 4-bit flits control field that may be used to specify information such as a virtual network for the flits, identify how the flits are to be encoded, and other examples. Additionally, error control functionality may be provided, such as through a 16-bit cyclic CRC field, as well as other potential examples.

[0068] The microchip format can be defined to optimize the throughput of messages on the link layer. Some traditional protocols have used smaller microchips without slots. For example, in QPI, 80-bit microchips are used. Although the throughput of larger (e.g., 192-bit microchips) may be lower, the message or packet throughput can be increased by optimizing the use of microchip data. For example, in some protocols, the entire microchip space (e.g., 80 bits) is utilized regardless of the message size or type. By subdividing larger microchips into slots of predetermined lengths and fields, the 192 microchip length can be optimized even when one or more of the available slots are not used sometimes, thereby achieving higher efficiency. In fact, it can be assumed that the link layer traffic includes many different types of messages and traffic, including messages and packets with varying header lengths and fields. The corresponding lengths and organization of the slots defined in the microchips can be defined to correspond to the statistical or expected frequencies of various messages and the needs of these messages. For example, two larger slots can be defined for each small slot to accommodate the expected statistical frequency of message delivery using these larger message types and header lengths, as well as other examples. In addition, flexibility can be provided, such as by floating payload fields, to further accommodate changing traffic volumes, such as Figure 6 In some instances, the microchip format may be fixed (including bits dedicated to specific slots in a microchip).

[0069] exist Figure 6 In the example of , an "Hdr" field may generally be provided for a microchip, and this field indicates a header indication of the microchip. In some instances, the Hdr field may indicate whether the microchip is a header microchip or a data microchip. In a data microchip, the microchip may still remain slotted, but the use of certain fields may be omitted or replaced with payload data. In some cases, the data field may include an opcode and payload data. In the case of a header microchip, various header fields may be provided. Figure 6 In the example of , an "Oc" field may be provided for each slot, the Oc field representing the operation code. Similarly, one or more slots may have a corresponding "msg" field, which represents the message type of the corresponding packet to be included in the slot (if the slot is designed to handle such a packet type, etc.). The "DNID" field may represent the destination node ID, the "TID" field may represent the transaction or tracker ID, the "RHTID" field may represent the requester node ID or the home tracker ID, and other potential fields. In addition, one or more slots may be provided with a payload field. In addition, in addition to other examples, a CRC field may be included in the microchip to provide a CRC value for the microchip.

[0070] The microslices divided into multiple slots of the GPIO protocol can be reused by the buffer memory protocol. Figure 7 A simplified block diagram 700 is shown illustrating an example topology of a computing system including CPU devices 705, 710 interconnected via GPIO interconnect links. Each CPU 705, 710 may be similarly connected to one or more corresponding buffer devices 715a-1 using a corresponding buffer memory protocol link ("MemLink"). Each buffer device may implement a memory controller for a system memory of the system. As described above, in some implementations, the buffer memory protocol interconnect may be based on the GPIO protocol because the physical layer and link layer of the buffer memory protocol are based on the same physical and link layer definitions of the GPIO protocol. Although Figure 7 Not shown, but the CPUs 705, 710 may further connect to one or more downstream devices using the GPIO protocol.

[0071] like Figure 7As further shown in the example of , the buffer device 715a-1 can be connected to a memory device such as a dual in-line memory module (DIMM) device. The memory corresponding to each buffer device can be considered to be local to the CPU (e.g., 705, 701) to which the buffer device is connected. However, other devices (including other CPUs) can access the memory through other slots using links that comply with the GPIO protocol. In some implementations, the port running the buffer memory protocol can only support commands for communicating with the memory and only support the buffer memory protocol (i.e., not the GPIO protocol and the buffer memory protocol). In addition, in some implementations, the GPIO interconnect protocol can support routing and indicate information such as request node identifiers and destination node identifiers (e.g., in its packets). On the other hand, the buffer memory protocol can be a point-to-point interface that does not utilize routing. Therefore, in packets sent using the buffer memory interface, some fields used in the GPIO protocol may be omitted. Alternatively, a field for carrying address decoding information host to the buffer can be specified, as well as other examples.

[0072] In other implementations, the buffer devices 715a-1 can support a two-level memory topology in which a certain amount of fast memory (e.g., DRAM) is used as a cache for larger, slower memory (e.g., non-volatile memory). In one such implementation, one or more of the buffer devices 715a-1 can use DDR as near, fast memory and transactional DDR DIMMs as larger "far" memory, among other examples. The transactional DIMMs can use a protocol (e.g., DDR-Transaction (DDR-T)) to communicate to volatile memory single inline memory modules (SIMMs) using transactional commands.

[0073] Buffered memory protocols and systems utilizing buffered memory protocols such as those shown above may be extended to enable implementation of a shared memory architecture that allows memory to be shared between independent nodes for exclusive or shared access using load / store (LD / ST) memory semantics. Fig. 8A , a simplified block diagram 800a illustrating an example system including a shared memory 805 that can be accessed by each of a plurality of independent nodes 810a-810n using load / store techniques is shown. For example, a shared memory controller 815 can be provided that can accept load / store access requests from various nodes 810a-810n on the system. The shared memory 805 can be implemented using synchronous dynamic random access memory (SDRAM), dual in-line memory modules (DIMMs), and other non-volatile memories (or volatile memories).

[0074] Each node itself may have one or more CPU sockets, and may also include local memory that is isolated from LD / ST accesses by other nodes in the system. The node may communicate with other devices on the system (e.g., shared memory controller 815, networking controller 820, other nodes, etc.) using one or more protocols (including PCIe, QPI, Ethernet, and other examples). In some implementations, a shared memory link (SML) protocol may be provided, through which low latency LD / ST memory semantics may be supported. For example, SML may be used to communicate reads and writes to shared memory 805 by various nodes 810a-810n of the system (through shared memory controller 815).

[0075] In one example, SML may be based on a memory access protocol such as Scalable Memory Interconnect (SMI) Generation 3 (SMI3). Other memory access protocols may be used instead, such as transactional memory access protocols (such as Fully Buffered DIMM (FB-DIMM), DDR Transactions (DDR-T), and other examples). In other instances, SML may be based on native PCIe memory read / write semantics with additional directory extensions. Memory protocol-based implementations of SML may provide bandwidth efficiency advantages because it is tailored for cache line memory accesses. Although there are high-performance inter-device communication protocols such as PCIe, the upper layers of such protocols (e.g., transaction layers and link layers) may introduce latency that degrades the application of a complete protocol for use with LD / ST memory transactions (including transactions involving shared memory 805). Memory protocols such as SMI3 may allow for the potential additional advantage of providing lower latency access because it may bypass a large portion of another protocol stack (such as PCIe). Therefore, implementations of SML may utilize SMI3 or another memory protocol running on the logical and physical PHY of another protocol, such as SMI3 on ​​PCIe.

[0076] As described above, in some implementations, a shared memory controller (SMC) 815 may be provided, which includes logic for processing load / store requests of nodes 810a-810n in the system. The SMC 815 may receive load / store requests by utilizing the SML and connecting nodes 810a-810n to links of the SMC 815. In some implementations, the SMC 815 may be implemented as a device, such as an application-specific integrated circuit (ASIC), that includes logic for servicing access requests of nodes 810a-810n for shared memory resources. In other instances, the SMC 815 (and the shared memory 805) may reside on a device, chip, or board that is separate from one or more (or even all) of the nodes 810a-810n. The SMC 815 may also include logic for coordinating transactions involving various nodes of the shared memory 805. In addition, the SMC may maintain directory tracking access to various data resources (such as each cache line) included in the shared memory 805. For example, a data resource may be in a shared access state (e.g., capable of being accessed (e.g., loaded or read) by multiple processing and / or I / O devices within a node simultaneously), an exclusive access state (e.g., reserved exclusively by a single processing and / or I / O device within a node, if not temporarily (e.g., for storage or write operations)), a non-cached state, and other potential examples. In addition, while each node may be able to directly access one or more portions of the shared memory 805, various nodes (e.g., 810a-810n) may employ different addressing schemes and values, resulting in the same shared memory data being referenced by a first node according to a first address value (e.g., in a first instruction), and a second node referencing the same data by a second address value. The SMC 815 may include logic that includes data structures that map addresses of nodes to shared memory resources to allow the SMC 815 to interpret various access requests by the various nodes.

[0077] In addition, in some cases, some portions of the shared memory (e.g., certain partitions, memory blocks, records, files, etc.) may be subject to certain permissions, rules, and allocations so that only a portion of the nodes 810a-810n are (e.g., SMC 815) allowed to access the corresponding data. In practice, each shared memory resource may be allocated to a corresponding (and in some cases, different) subset of the nodes 810a-810n of the system. These allocations may be dynamic, and SMC 815 may (e.g., on demand, dynamically, etc.) modify such rules and permissions to accommodate new or changed rules, permissions, node allocations, and ownership that may be applied to a given portion of the shared memory 805.

[0078] The example SMC 815 can also track various transactions involving nodes (e.g., 810a-810n) in the system accessing one or more shared memory resources. For example, the SMC 815 can track the information of each shared memory 805 transaction, including the identification of (one or more) nodes involved in the transaction, the progress of the transaction (e.g., whether it has been completed), and other transaction information. This can allow some of the transaction-oriented aspects of the traditional distributed memory architecture to be applied to the improved multi-node shared memory architecture described herein. In addition, transaction tracking (e.g., performed by the SMC) can be used to assist in maintaining or enforcing different and independent fault domains of each corresponding node. For example, the SMC can maintain the corresponding node ID for each ongoing transaction in its internal data structure (including in the memory), and use this information to enforce access rights and maintain separate fault domains for each node. Thus, when one of the nodes fails (e.g., due to a critical error, a triggered recovery sequence, or other failure or event), only that node and its transactions involving shared memory 805 are interrupted (e.g., abandoned by the SMC) - the remaining nodes' transactions involving shared memory 805 continue independently of the failure in the other node.

[0079] The system may include multiple nodes. In addition, some example systems may include multiple SMCs. In some cases, a node may be able to access shared memory outside of a remote SMC to which it is not directly attached (i.e., the node's local SMC is connected to the remote SMC via one or more SML link hops). The remote SMC may be located on the same board or on a different board. In some cases, some nodes may be outside the system (e.g., off-board or off-chip), but still have access to shared memory 805. For example, among other examples, one or more off-system nodes may be directly connected to the SMC using a link that complies with the SML. In addition, other systems that include their own SMCs and shared memory may also be connected to the SMC 810 to extend sharing of memory 805 to nodes included therein (e.g., on another board that interfaces with another SMC that is connected to the SMC via an SML link). In addition, network connections may be tunneled to further extend access to other off-board or off-chip nodes. For example, the SML may be tunneled to couple communication grounds to the SMC. Fig. 8A The example system of may provide an Ethernet connection (eg, provided by network controller 820) to another system that may also include one or more other nodes and allow those nodes to also gain access to SMC 815 and therefore shared memory 805, as well as other examples.

[0080] As another example, as shown in the simplified block diagram 800b of FIG8b, an improved shared memory architecture that permits shared access by multiple independent nodes according to LD / ST memory semantics can flexibly allow a variety of different multi-node system designs to be provided. Various combinations of multiple nodes can be assigned to share portions of one or more shared memory blocks provided in the example system. For example, Figure 8B Another example system shown in the example of may include multiple devices 850a-850d implemented as, for example, separate dies, boards, chips, etc., each device including one or more independent CPU nodes (e.g., 810a-810h). Each node may include its own local memory. One or more of the multiple devices 850a-850d may also include a shared memory that can be accessed by two or more of the nodes 810a-810h of the system.

[0081] Figure 8BThe illustrated system is provided to illustrate some of the variability that can be achieved by an improved shared memory architecture such as shown and described herein. For example, each of device A 850a and device C 850c may include a corresponding shared memory element (e.g., 805a, 805b). Therefore, in some implementations, each shared memory element on different devices may also include a corresponding shared memory controller (SMC) 815a, 815b. Various combinations of nodes 810a-810h may be communicatively coupled to each SMC (e.g., 815a, 815b) to allow nodes to access corresponding shared memories (e.g., 805a, 805b). As an example, the SMC 815a of device A 850a may be connected to nodes 810a, 810b on device A using a direct data link supporting SML. In addition, another node 810c on another device (e.g., device C 850c) can also access the shared memory 805a by means of a (SML-enabled) direct hardwired connection from the node 810c (and / or its device 850c) to the SMC 815a. Indirectly, a network-based connection or other such connection can also be used to allow nodes (e.g., 810f-810h) of remote or off-board devices (e.g., device D 850d) to interface with the SMC 815a using a conventional protocol stack, thereby also accessing the shared memory 805a. For example, the SML tunnel 855 can be established through Ethernet, InfiniBand, or other connections coupling device A and device D. Although establishing and maintaining the tunnel may introduce some additional overhead and latency compared to SML running over other less software-managed physical connections, the SML tunnel 855, when established, can be used as other SML channels and allow nodes 810f-810h to interface with SMC 815a via the SML and access shared memory 805a, as any other node communicating with the SMC via an SML link can. For example, reliability and ordering of packets in the SML channel can be enforced by networking components in the system, or it can be enforced end-to-end between SMCs.

[0082] In still another example, a node (e.g., 815d, 815e) on a device different from the device hosting a particular portion of the shared memory (e.g., 805a) can be indirectly connected to a corresponding SMC (e.g., SMC 815a) by directly connecting to another SMC (e.g., 815b), which itself is coupled to the corresponding SMC (e.g., 815a) (e.g., using an SML link). Linking two or more SMCs (e.g., 815a, 815b) can effectively expand the amount of shared memory available to nodes 810a-810h on the system. For example, by means of the link between SMCs 815a, 815b in the example of FIG. 8b, in some implementations, any of the nodes (e.g., 810a-810c, 810f-810h) that can access the shared memory 805a through SMC 815a can also potentially access the sharable memory 805b by means of the connection between SMC 815a and SMC 815b. Similarly, in some implementations, each node that directly accesses SMC 815b may also access sharable memory 805a via the connection between SMCs 815a, 815b, among other potential examples.

[0083] As described above, independent nodes may each access shared memory (including shared memory included in memory that is not connected to an SMC to which the node is directly connected). The shared memory is effectively pooled. While traditional buffered memory protocols may employ point-to-point communications, the pooling of shared memory and the joint management of this memory by multiple SMCs may involve packets and microslices associated with this memory traversing multiple hops and SMCs before they reach their intended destination. In this regard, multiple SMCs may form a network of SMCs, and each SMC may include logic for determining how to route a particular microslice from its directly connected node to the SMC to which the memory addressed by the microslice is connected. For example, in Figure 8CIn the embodiment of the present invention, an example 800c in which multiple SMCs 815a-c are interconnected with each other SMC through one or more SML links. Each SMC can be connected to a subset of the processor nodes in the system. In addition, each SMC can be directly connected to a corresponding subset of the memory elements constituting the shared memory pool and provide access to the corresponding subset. As an example, SMC 815a can be connected to nodes 810a, 810b and a shared memory element (e.g., 805a). Another node 810i can access the memory row stored in the shared memory portion 805a by sending a request to SMC 815c (which can route the request to SMC 815a through an SML link). SMC 815a can manage the memory actions related to the request, and in some cases respond by providing read data, a reply or other information to node 810i by routing the response to SMC 815c through an SML link. As an alternative to routing SML communications directly between SMCs 815a and 815c, in other instances, communications may be routed through other SMCs (e.g., 815b) in addition. Thus, each SMC in a shared memory architecture may include routing logic implemented in hardware and / or software to facilitate routing communications between SMCs within the network.

[0084] As described above, the improved shared memory architecture may include a low latency link protocol (i.e., SML) based on a memory access protocol such as SMI3, and is provided to facilitate load / store requests involving shared memory. Whereas conventional SMI3 and other memory access protocols may be configured for use with memory sharing within a single node, SML may extend memory access semantics to multiple nodes to allow memory sharing between multiple nodes. In addition, SML may potentially be used for any physical communication link. SML may utilize a memory access protocol that supports LD / ST memory semantics, which is overlaid on a physical layer (and corresponding physical layer logic) adapted to interconnect different devices (and nodes). In addition, the physical layer logic of SML may provide no packet loss and error retry functionality and other features.

[0085] In some implementations, SML can be implemented by overlaying SMI3 on ​​the PCIe PHY. An SML link layer can be provided (e.g., in place of a traditional PCIe link layer) to discard flow control and other features and facilitate memory access with lower latency as would be typical in a traditional CPU memory access architecture. In one example, the SML link layer logic can be multiplexed between shared memory transactions and other transactions. For example, the SML link layer logic can be multiplexed between SMI3 and PCIe transactions. For example, SMI3 (or another memory protocol) can be overlaid on top of PCIe (or another interconnect protocol) so that the link can dynamically switch between SMI3 and PCIe transactions. This can allow traditional PCIe traffic to effectively coexist on the same link as SML traffic in some instances.

[0086] Go to Fig. 9 , a representation 900 illustrating a first implementation of SML is shown. For example, SML can be implemented by overlaying SMI3 on ​​a PCIe PHY. The physical layer can use standard PCIe 128b / 130b encoding for all physical layer activities (including link training) as well as PCIe data blocks. SML can provide traffic on the lanes of the link (e.g., lane 0-lane 7) to be multiplexed between PCIe packets and SMI3 microslices. For example, in Fig. 9 In the illustrated implementation, the PCIe 128b / 130b encoded synchronization header may be modified and used to indicate that SMI3 flits are to be sent on lanes of a link rather than PCIe packets. In traditional PCIe 128b / 130b encoding, a valid synchronization header (e.g., 910) may include the sending of a 10b pattern on all lanes of the link (indicating that the payload type of the block should be a PCIe data block) or the sending of a 01b pattern on all lanes of the link (indicating that the payload type of the block should be a PCIe ordered set block). In one example of an SML, an alternative synchronization header may be defined to distinguish SMI3 flits traffic from PCIe data blocks and ordered sets. Fig. 9 In one example shown in FIG. 9 , a PCIe 128b / 130b sync header (e.g., 905a, 905b) may be encoded with an alternating 01b, 10b pattern on odd / even lanes to identify that an SMI3 microflake is to be transmitted. In another alternative implementation, a 128b / 130b sync header encoding for SMI3 traffic may be defined by alternating 10b, 01b patterns on odd / even lanes, as well as other example encodings. In some cases, an SMI3 microflake may be transmitted immediately after the SMI3 sync header on a per-byte basis, with the transition between the PCIe and SMI3 protocols occurring at block boundaries.

[0087] In some implementations, such as Fig. 9 In implementations such as those shown in the examples of , the transition between protocols can be defined to occur at a block boundary, regardless of whether it corresponds to an SMI3 microchip or a PCIe packet boundary. For example, a block can be defined to include a predefined amount of data (e.g., 16 symbols, 128 bytes, etc.). In this implementation, when a block boundary does not correspond to an SMI3 microchip or a PCIe packet boundary, the transmission of the entire SMI3 microchip can be interrupted. The interrupted SMI3 microchip can be restored in the next SMI3 block indicated by the transmission of another synchronization header for SMI3 encoding.

[0088] Go to Fig. 10A , a representation 1000 illustrating another example implementation of SML is shown. Fig. 10A In the example of , instead of using specialized synchronization header encoding to signal the transition between memory access and interconnect protocol traffic, a physical layer framing token may be used. A framing token (or "token") may be a physical layer data encapsulation that specifies or implies the number of symbols to be included in the data stream associated with the token. Thus, a framing token may identify that a stream is beginning and implies where it will end, and thus may be used to also identify the location of the next framing token. The framing token for a data stream may be located in the first symbol (symbol 0) of the first lane (e.g., lane 0) of the first data block of the data stream. In the example of PCI, five framing tokens may be defined, including a Start of TLP Traffic (STP) token, an End of Data Stream (EDS) token, an End of Bad (EDB) token, a Start of DLLP (SDP) token, and a Logical Idle (IDL) token.

[0089] exist Fig. 10A In some examples, SML can be implemented by overlaying (or "tunneling") SMI3 or another data access protocol over PCIe, and the standard PCIe STP token can be modified to define a new STP token that identifies that SMI3 (rather than TLP traffic) is to be initiated on a channel of the link. In some examples, the value of a reserved bit of the standard PCIe STP token can be modified to define an SMI3 STP token in the SML. In addition, as Fig. 10BAs shown, the STP token 1005 may include several fields, including a 1010 field, which identifies that what is to follow is the length of the SMI3 payload (in terms of the number of microchips). In some implementations, one or more standard payload lengths may be defined for TLP data. In some implementations, the SMI3 data may be defined to include a fixed number of microchips, or in other cases may have a variable number of microchips (in this case, the length field for the number of SMI3 microchips becomes a field that can be ignored). In addition, the length field of the SMI3 STP may be defined as a length different from one of the defined TLP payload lengths. Therefore, as an example, an SMI3 STP may be identified based on a non-TLP length value present in the STP length field. For example, in one implementation, the upper 3 bits in the 11-bit STP length field may be set to 111b to indicate an SMI3 packet (e.g., based on the assumption that a PCIe TLP that does not comply with the specification may be long enough to have a length field with the upper 3 bits of the length field being 1). Other implementations may alter or encode other fields of the STP token to distinguish a PCIe STP token that identifies a legacy PCIe TLP data payload from an SMI3 STP token that identifies that SMI3 data is encapsulated within the TLP data.

[0090] Back to Fig. 10A In an example of , the sync header data may follow the encoding specified for traditional PCIe 128b / 130b encoding. For example, at 1015a-c, a sync header with a value of 10b is received, indicating that a data block is coming. When a PCIe STP (e.g., 1020) is received, a PCIe TLP payload is expected and the data stream is processed accordingly. Consistent with the payload length identified in the PCIe STP 1020, the PCIe TLP payload may utilize the full payload length allocated. Following the end of the TLP payload, another STP token may be received at substantially any time within the data block. For example, at 1025, an SMI3 STP may be received that signals a conversion from PCIe TLP data to SMI3 microchip data. For example, the SMI3 STP may be sent once the end of the PCIe packet data is identified.

[0091] continue Fig. 10AFor example, as with PCIe TLP data, the SMI3 STP 1025 can define the length of the SMI3 microslice payload to follow. For example, the payload length of the SMI3 data can correspond to the number of SMI3s in the DW to follow. Thus, a window corresponding to the payload length can be defined on the channel (e.g., ending at symbol 15 of channel 3), wherein only SMI3 data is to be sent during the window. When the window ends, other data can be sent, such as another PCIe STP to restart the sending of TLP data or other data (such as ordered set data). For example, if Fig. 10A As shown in the example of , the EDS token is sent following the end of the SMI3 data window defined by the SMI3 STP token 1025. The EDS token can signal the end of the data flow and hint that an ordered set block will follow, such as Fig. 10A . A synchronization header 1040 encoded with 01b is sent to indicate that an ordered set block is to be sent. In this case, a PCIe SKP ordered set is sent. Such ordered sets can be sent periodically or according to set intervals or windows so that various PHY level tasks and coordination can be performed, including initialization bit alignment, initialization symbol alignment, exchanging PHY parameters, compensating for different bit rates of two communication ports, and other examples. In some cases, a mandatory ordered set can be sent to interrupt the data block or defined window specified by the corresponding SMI3 STP token for SMI3 micro-slice data.

[0092] Although in Fig. 10A Although not explicitly shown in the example of , STP tokens may also be used for conversion from SMI3 flits data on a link to PCIe TLP data. For example, following the end of a defined SMI3 window, a PCIe STP token (e.g., similar to token 1020) may be sent to indicate that the next window is for sending a specified amount of PCIe TLP data.

[0093] In some embodiments, memory access microchips (e.g., SMI3 microchips) may vary in size, which makes it difficult to predict in advance how much data is to be retained in the corresponding STP token (e.g., SMI3 STP token) for the memory access payload. As an example, as shown in FIG. 10 , an SMI3 STP 1025 may have a length field indicating that 244 bytes of SMI3 data are expected to follow the SMI3 STP 1025. However, in this example, only 10 microchips (e.g., SMI3 microchips 0-9) are ready to be sent during the window, and these 10 SMI3 microchips only utilize 240 bytes of the 244 bytes. Therefore, four (4) bytes of empty bandwidth remain, and these bytes are filled with IDL tokens. This can be particularly suboptimal when PCIe TLP data is queued and waiting for the SMI3 window to close. In other cases, the window provided for sending the SMI3 microchip may be insufficient to send the amount of SMI3 data ready for the channel. Arbitration techniques can be used to determine how to arbitrate between SMI3 and PCIe TLP data that coexist on a link. In addition, in some implementations, the length of the SMI3 window can be dynamically modified to help use the link more efficiently. For example, arbitration or other logic can monitor how well the defined SMI3 window is utilized to determine whether the defined window length can be better optimized to the amount of SMI3 (and competing PCIe TLP traffic) expected for the channel. Therefore, in such an implementation, the length field value of the SMI3 STP token can be dynamically adjusted (e.g., between different values) depending on the amount of link bandwidth to which the SMI3 microchip data should be allocated (e.g., relative to other PCIe data (including TLP, DLLP, and ordered set data)) and other examples.

[0094] Go to Fig.11 , illustrates a representation 1100 of another example implementation of SML. In this alternative embodiment, SML can provide interleaved SMI3 and PCIe protocols through modified PCIe framing tokens. As described above, the EDS token can be used in PCIe to indicate the end of a data stream and indicate that the next block will be an ordered set block. Fig.11In an example of , the SML may define an SMI3 EDS token (e.g., 1105) indicating the end of a TLP data stream and a transition to an SMI3 microchip transfer. The SMI3 EDS (e.g., 1105) may be defined by encoding a portion of the reserved bits of a traditional EDS token to indicate that SMI3 data will follow after the PCIe EDS instead of a PCIe ordered set or other data. Unlike traditional EDS tokens, the SMI3 EDS may be sent substantially anywhere within a PCIe data block. This may allow for additional flexibility in sending SMI3 data and accommodating corresponding low-latency shared memory transactions. For example, a transition from PCIe to SMI3 may be implemented with an overhead of a single double word (DW). In addition, like traditional EDS tokens, the example SMI3 EDS may not specify a length associated with the SMI3 data to follow the token. After the SMI3 EDS, the PCIe TLP data may end and the SMI3 microchip may proceed on the link. SMI3 traffic may proceed until the SMI3 logic passes control back to the PCIe logic. In some implementations, the sending of an SMI3 EDS results in control being passed from the PCIe logic to SMI3 logic provided, for example, on a device connected on the link.

[0095] In one example, SMI3 (or another protocol) may define its own link control signaling for use in performing link layer control. For example, in one implementation, SML may define a specialized version of the SMI3 Link Layer Control (LLCTRL) microflake (e.g., 1110) that indicates a transition from SMI3 back to PCIe protocol. As with the SMI3 EDS, the defined LLCTRL microflake (e.g., 1110) may cause control to be transferred from the SMI3 logic back to the PCIe logic. In some cases, such as Fig.11 As shown in the example of , before completing the transition to PCIe, the defined LLCTRL microtip (e.g., 1110) can be filled with a predefined number of LLCTRL idle (LLCTRL-IDLE) microtip (e.g., 1115). For example, the number of LLCTRL-IDLE microtip 1115 to be sent to fill the SMI3 LLCTRL microtip 1110 can depend on the latency used to decode the defined SMI3 LLCTRL microtip 1110 that signals the transition. After completing the transition back to PCIe, the STP packets can be sent and the TLP packet data on the link can be restarted under the control of PCIe.

[0096] As combined Figures 8A-8CAs described in the example of , a shared memory controller in a shared buffer memory architecture may include routing logic implemented in hardware and / or software to facilitate data transactions involving nodes and shared memory by routing communications through SML links between SMCs. To facilitate routing of microslices among an SMC network, additional fields may be provided in the SML microslices to help route requests and responses to those requests to their appropriate destinations. Table 1 describes example fields that may be provided in microslices to accommodate routing in a multi-SMC shared memory architecture.

[0097] Table 1: Alternative fields for SML microsheets

[0098] PLM Grouping Field Increment (δ) describe Source SMC link ID[8:0] Encodes the source SMC and internal SMC proxy for the request Address [59:52] High bit of the global system address Slot 2 Mode Bit When set, indicates that a particular slot in the defined multi-slot flit format has changed its intent to carry some of the upper address bits. When cleared, the particular slot carries information according to the buffer memory link protocol. Alternative Routing Requested (ARR) Specifies that the request should take an alternative route to its destination via the construct

[0099] As introduced in Table 1, in some implementations, a microchip may be provided with a field for indicating the source (and in some cases, the destination) node identifier of a given transaction. The link ID field may be encoded with a bit that identifies the identity of the corresponding SMC, as well as some additional bits for identifying the specific link (or SMC agent) on which the SMC sends / receives the request. The SMC identifier may be a unique value within the system. In instances where the link ID value is based on a combination, combination, or other manner of the SMC's ID and the ID of a specific link, it is expected that the value of the link ID field is globally unique within the system. In the following implementations, a similar field may be provided to indicate the destination SMC of the request (although in other implementations, the destination SMC may be identifiable from the memory address that is the subject of the transaction). In addition, transactions generated internally by a specific SMC for events such as memory migration, memory read / write / clear related to reliability, accessibility, and serviceability (RAS), etc. may have one or more unique "link numbers", even if they do not reside outside the external link. The Link ID field may be processed and interpreted by the receiving SMC to determine how the flits are to be routed to / from their destination either locally (between a directly connected node and the SMC) or through one or more additional SMCs and one or more SML "hops".

[0100] By pooling the memory of multiple memory elements managed by multiple different SMCs, the size of the address space can be significantly larger than the size that would be expected for point-to-point transactions between nodes and a single corresponding memory controller. For a pooled memory architecture, a larger address space can be made available to support the needs of all nodes (e.g., in a rack or server tower), which will be the total amount of memory for each node in the pool. In some cases, the private memory of each node may reside outside the directly attached SMC, which is not a requirement, and the dedicated memory of the node and part of the shared memory may be located outside the remote SMC. In some implementations, the microchip format of the existing protocol can be reused and expanded to support a shared memory infrastructure. In fact, the system can utilize the same basic microchip format for point-to-point buffer memory links and shared memory links.

[0101] In one example, to accommodate the larger address spaces involved in a shared memory architecture, additional bits may be provided beyond and above the bits provided in the defined flit header format for GPIO and / or standard buffered memory access transactions. For example, the slots of a multi-slot flit may be used to encode additional address bits and expand the memory address scheme supported by the flit format (i.e., in SML transactions). For example, Fig.12 FIG. 10 illustrates an example high performance GPIO interconnect protocol microchip header format (such as, in conjunction with Figure 6 1200 of an amplified version of the example format shown and described. For a standard buffered memory link interface, the payload of slots 0 and 1 can be used to encode the address of the row in memory to be accessed in the transaction (e.g., a 64-byte cache line address). In one implementation, additional address bits can be encoded in the bits of slot 2 (i.e., in UI 6, 7, and 8) as additional bits or high address bits of the pooled memory space. This can extend the address space by eight bits.

[0102] continue Fig.12 For example, Figure 6 The three-slot microchip format of the GPIO protocol described in can be augmented with transaction-specific fields for shared memory architectures (including fields for facilitating routing between multiple SMCs in such an architecture). In some implementations, the upper address bits of the extended memory address can also be used to replace the SMC number of the "Source Link ID" field, such as when there is a flat system address space, when the upper physical address bits represent the SMC number, and other examples.

[0103] As an additional example, the microchip format of the GPIO protocol or buffered memory link protocol may be augmented to accommodate additional bits to optionally indicate the alternate route(s) to be taken if multiple paths exist between a source-destination pair. Fig.12In an example, a reserved field or a floating field provided in the header microchip format of the GPIO protocol may be encoded with an ARR value for each of two transactions (in slots 0 and 1) to indicate a specific path to be taken between the SMCs of the shared memory architecture. For example, the ARR value may be used to provide high availability within a system. For example, an SMC or a node to which it is connected may detect a timeout condition for a transaction. This may prompt a retry using an available alternative path to route the transaction between the SMC and the destination SMC (e.g., after one or more failed retries using the original routing path). To facilitate this, the SMC may encode bits of the ARR field of the transaction with a value indicating an alternative path. This may allow the transaction to be retried along (one or more) alternative paths to recover from an error without escalating the error to an unrecoverable (such as a timeout or poisoning).

[0104] As described above, the microchip of the SML protocol can be based on a microchip having a basic format defined by another interconnect protocol (e.g., another protocol used in the system) or as an amplified version of the microchip. In some instances, modifications to the basic format can be selected: in some cases the original basic microchip / packet layout (of another protocol) can be retained, and an amplified version can be applied to others. For example, a microchip can utilize one or more bits to indicate whether the microchip is according to the basic microchip format or a modified microchip format. The receiving SMC (or node) can decode the microchip to determine which format is applied to the microchip. In one example, such as Fig.12 As shown, a bit may be provided to indicate whether slot 2 of the basic microchip format is to be repurposed to provide extended memory address bits and / or source node ID bits, among other examples. In one example, where only a single virtual network (e.g., VN0) is to be used, this bit may occupy the defined virtual network bits. In fact, the bit used to indicate how slot 2 is to be used may itself represent a modification to the basic microchip format, among other examples. In one example, the mode bit may be reused (e.g., in Fig.12 In the example row UI0), slot 2 is used to specify whether it is to be used for extended address or basic microchip format, as well as other examples.

[0105] As described above, in some implementations, a microchip format may be provided that indicates both the source SMC and the destination SMC identifiers. In some alternative embodiments, only the source SMC identifier (or "node ID") may be provided, and the destination SMC may be implicitly identified from the address bits provided to indicate the row of the corresponding memory involved in the transaction. In other words, the SMC may determine that a specific memory row referenced in the address field of the microchip corresponds to a memory accessed by a specific SMC among multiple SMCs in a shared memory architecture. For example, a record may be maintained at each SMC that indicates which addresses correspond to which memory elements directly connected to which SMC. In other implementations, high-order address bits (e.g., bits added in the extension of the microchip address field) may correspond to a specific SMC within the shared memory architecture. Therefore, the destination SMC (connected to the target memory address) may be identified by another SMC from the extended memory address field of the microchip. The microchip may be decoded by each SMC at each "jump" of the route requesting the microchip to its destination. Thus, each SMC can identify the destination SMC from the address bits and determine the next hop that allows the request to proceed toward the destination SMC.

[0106] A processor node in a shared memory architecture that employs multiple different SMCs (each SMC managing a subset of pooled and / or shared system memory) may be unaware of the architecture used to access a requested memory row from its directly connected (or "local") SMC. For example, a processor node may not be aware of its local SMC, which determines that another SMC manages the requested portion of memory and forwards the node's request for processing by the other SMC. All nodes see that it sends a request to its local SMC and receives a corresponding response. The node may also be unaware of the fact that the requested memory row is in pooled memory. Therefore, in some cases, a node may send a micro-slice corresponding to a request involving a particular memory row, and send the micro-slice according to a single-node buffered memory link protocol (e.g., without the enhancement fields provided for the shared memory link protocol followed between SMCs). In such instances, at the ingress port of the local SMC, an additional enhancement field (such as that in Table 1 or 2) may be added before the micro-slice (or packet) is sent out within the SMC. Fig.12 The enhanced fields (shown in the example of ) are attached to a microtile (or packet). At egress (e.g., to a processor node), these enhanced fields can be stripped off.

[0107] In some cases, processor nodes may not be equipped to correctly track their transactions in view of the potential complexity that multiple jumps between SMCs in a shared memory architecture can be hidden from processor nodes. Therefore, the local SMC can track transactions on behalf of its corresponding (one or more) processor nodes. The local SMC of the node can save all routing-related fields included in the original micro slice of the request (e.g., fields included in the micro slice formatted according to the buffer memory link protocol of the node (which may be different from the SML protocol)). As the SMC can then be responsible for a potential large number of transactions, the memory of the SMC itself can be allocated to accommodate the tracking of transaction information. In some instances, the SMC can optimize the number of bits tracked for each transaction by allocating or converting the fields of the original request to values ​​with fewer bits than the original bits. For example, the SMC can allocate a RHTID field value at the SMC level, which has a bit that is assumed to be less than the original bit. Transaction information can be stored in an on-chip (i.e., on the SMC) storage with CAM lookup capability to retrieve the original bits at the exit. On-chip memory may be supplemented with (or alternatively replaced by) local memory (eg, cache memory managed by the SMC) as storage above and beyond that available on-chip memory, among other examples.

[0108] It should be understood that the implementations described herein are provided as examples to illustrate certain principles and features disclosed in this specification. It should be understood that alternative configurations, protocols, and architectures (different from those specifically discussed in the examples) can utilize and apply such principles and features. As an example of an alternative, PCIe memory read / write (e.g., instead of the SMI3 protocol) enhanced with directory information can be used. The directory information can be implemented by the reserved bits of the PCIe packet. In another example, the CPU node can send memory read / write transactions on the PCIe link using a cache controller (e.g., as an alternative to a shared memory controller), for example, based on remote address range checks and other potential examples and alternatives.

[0109] Go to Figures 13A-13B , flow charts 1300a-b are shown, which illustrate example techniques for communicating using a shared memory link interconnect. Fig.13AIn the embodiment of the present invention, a load / store memory access message may be received 1305 from a processor node at one of a plurality of shared memory controllers (SMCs) (e.g., via an SML link) requesting data at a specific address of a shared memory. The message may be embodied in one or more microslices (including a header microslice for identifying a specific address). The shared memory controller may determine 1310 from the specific address (or a destination node identifier identifying a specific one of the SMCs) that the message (or "request") is to be processed by a specific SMC because a specific shared memory element of the specific address is hosted in a shared memory element in a specific SMC management system. In some cases, the SMC directly connected to the processor node is the specific SMC. In other cases, the specific SMC is another of the plurality of SMCs. The SMC may determine whether it or another SMC is to process the request (e.g., at 1315). In the case where the SMC determines that another SMC is the specific SMC for processing the request, the SMC may determine 1320 how to route the request to the specific SMC. In some cases, this may involve sending the request to an intermediate SMC among the plurality of SMCs. In other cases, the SMC may determine that routing involves sending the request directly (e.g., over a single SML link) to a specific other SMC. The SMC then sends 1325 the request (e.g., over an SML link) to another SMC (a specific SMC or an intermediate SMC). In some implementations, the format of the microslices used by the processor node in its request may be augmented by the SMC to facilitate routing of microslices within the SMC network. For example, the SMC may extend the address indicated by the processor node with one of multiple slots, add a source node identifier or a destination node identifier to indicate the source SMC or destination SMC in the transaction, encode bits to indicate that specific microslice slots are being used for these purposes, and other modifications.

[0110] Once a particular SMC accesses a particular row of shared memory corresponding to a particular address, the particular SMC may send a response back to the SMC connected to the particular node. The response may include enhanced fields (such as those included in the request microslice) to help route the response back to the source SMC (e.g., source node ID). After routing back along the same or different path within the SMC network, the SMC connected to the particular node may receive 1330 the response generated by the particular SMC, and may provide 1335 the response to the processor node. In the case where the microslice format used between SMCs represents an amplified version of the microslice consumed by the processor node, the SMC may strip the "extra" or enhanced fields from the response microslice before providing 1335 the response to the processor node. For the processor node, the response may appear to be completely processed by the SMC to which it is connected. In other words, the node may be unaware of the fact that the memory row is managed by another SMC and the request is routed through one or more other SMCs in the SMC network in the shared memory architecture.

[0111] Where the SMC determines (eg, at 1315) that it manages a memory element that hosts the requested memory line, the SMC may access 1340 the particular memory line from the memory element, generate a response 1345, and provide 1335 the response to the processor node.

[0112] exist Fig. 13BIn the example of , the SMC may receive 1350 a microchip from another SMC associated with the routing of the microchip between the source SMC and the destination SMC. The source SMC is connected to a processor node responsible for generating a memory (e.g., load / store) request involving a specific row of shared memory at a specific address. The destination SMC is connected to a memory element that hosts a specific row of shared memory. The destination SMC generates a response to the request when accessing the specific row of shared memory. The microchip may include fields such as an extended address field, a source node ID for identifying the source SMC, a destination node ID for identifying the destination SMC, an alternative routing path field for indicating a specific routing path in multiple available routing paths in the SMC network, and other fields. These fields may be included in the request microchip and the response microchip. In addition, these fields may be added to expand the basic microchip format, such as a microchip format defined according to a single node buffer memory access protocol, a GPIO protocol, or another protocol. Each SMC in the routing path may decode 1355 a microchip to determine the next hop in 1360 routing. For example, in the case where the microchip represents a request, the microchip may be decoded to identify the extended address of a specific memory row. The SMC can determine the destination SMC from the extended address and determine how to forward the request to the destination SMC (e.g., directly or through which intermediate SMC). In other instances, the SMC may alternatively (or additionally) consult the destination node ID field of the microchip. In the case where the microchip embodies at least a portion of the response to the request (generated by the destination SMC), the SMC can decode the 1355 microchip to identify the source node ID corresponding to the source SMC and determine the next step in the corresponding routing path to route the response microchip to the source SMC for delivery to the processor node that initiated the transaction. In some cases, this return routing path can be the same (but reverse) path used to send the request from the source SMC to the destination SMC. In other cases, an alternative path can be used to send the response from the source SMC to the target SMC.

[0113] It should be noted that although many of the above principles and examples are described in the context of the specific revision of PCIe and PCIe specifications, the principles, solutions and features described herein can be equally applicable to other protocols and systems. For example, similar channel errors can be detected in these other links by using other protocols based on similar symbols, data streams and tokens and for the use, placement and formatting of such structures in the data transmitted by other links. In addition, alternative mechanisms and structures (for example, except PCIe LES registers or SKP OS) can be used to provide channel error detection and reporting functions in the system. In addition, the combination of the above solutions can be applied in the system, including the combination of the logic and physical enhancement of the link and its corresponding logic as described herein, and other examples.

[0114] Note that the above-described apparatus, method, and system can be implemented in any of the aforementioned electronic devices or systems. As a specific illustration, the following figures provide exemplary systems for utilizing the present invention as described herein. As the following systems are described in more detail, many different interconnections according to the above discussion are disclosed, described, and revisited. And as is readily apparent, the above-described advances can be applied to any of these interconnections, configurations, or architectures.

[0115] refer to Fig.14 , depicts an embodiment of a block diagram of a computing system including a multi-core processor. Processor 1400 includes any processor or processing device, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a handheld processor, an application processor, a coprocessor, a system on a chip (SOC), or other device for executing code. In one embodiment, processor 1400 includes at least two cores, core 1401 and core 1402, which may include asymmetric cores or symmetric cores (the illustrated embodiment). However, processor 1400 may include any number of processing elements, which may be symmetric or asymmetric.

[0116] In one embodiment, a processing element refers to hardware or logic that supports a software thread. Examples of hardware processing elements include: thread units, thread slots, threads, processing units, contexts, context units, logical processors, hardware threads, cores and / or any other elements that can preserve the state of a processor, such as an execution state or an architecture state. In other words, a processing element, in one embodiment, refers to any hardware that can be independently associated with a code (such as a software thread, an operating system, an application, or other code). A physical processor (or processor socket) generally refers to an integrated circuit that potentially includes any number of other processing elements, such as cores or hardware threads.

[0117] A core often refers to logic located on an integrated circuit that can maintain an independent architectural state, wherein each independently maintained architectural state is associated with at least some dedicated execution resources. In contrast to a core, a hardware thread generally refers to any logic located on an integrated circuit that can maintain an independent architectural state, wherein the independently maintained architectural states share access to execution resources. As can be seen, when certain resources are shared and other resources are dedicated to an architectural state, the lines between the nomenclature of a core and a hardware thread overlap. However, cores and hardware threads are viewed as separate logical processors by the operating system, wherein the operating system can schedule operations on each logical processor separately.

[0118] like Fig.14The illustrated physical processor 1400 includes two cores, cores 1401 and 1402. Here, cores 1401 and 1402 are considered symmetric cores, i.e., cores with the same configuration, functional units and / or logic. In another embodiment, core 1401 includes an out-of-order processor core, and core 1402 includes an in-order processor core. However, cores 1401 and 1402 may be individually selected from any type of core, such as a native core, a software management core, a core suitable for executing a native instruction set architecture (ISA), a core suitable for executing a converted instruction set architecture (ISA), a co-designed core, or other known cores. In a heterogeneous core environment (i.e., an asymmetric core), a certain form of conversion, such as binary conversion, may be utilized to schedule or execute code on one or both cores. However, in order to advance the discussion, the functional units illustrated in core 1401 are described in more detail below, because the units in core 1402 operate in a similar manner in the depicted embodiment.

[0119] As depicted, core 1401 includes two hardware threads 1401a and 1401b, which may be referred to as hardware thread slots 1401a and 1401b. Therefore, a software entity (such as an operating system) in one embodiment potentially views processor 1400 as four separate processors, i.e., four logical processors or processing elements capable of executing four software threads simultaneously. As mentioned above, a first thread is associated with architecture state register 1401a, a second thread is associated with architecture state register 1401b, a third thread is associated with architecture state register 1402a, and a fourth thread is associated with architecture state register 1402b. Here, each architecture state register (1401a, 1401b, 1402a, and 1402b) may be referred to as a processing element, thread slot, or thread unit, as described above. As illustrated, architecture state register 1401a is replicated in architecture state register 1401b, so that individual architecture states / contexts can be stored for logical processor 1401a and logical processor 1401b. In core 1401, other smaller resources, such as renaming logic and instruction pointers in allocator and rename block 1430, may also be replicated for threads 1401a and 1401b. Some resources, such as reorder buffers, ITLB 1420, load / store buffers, and queues in reorder / retirement unit 1435, may be shared through partitioning. Other resources, such as general internal registers, page table base registers, low-level data cache and data TLB 1415, execution units 1440, and portions of out-of-order units 1435, are potentially fully shared.

[0120] Processor 1400 often includes other resources, which may be fully shared, shared through partitioning, or dedicated by / to processing elements. Fig.14, an embodiment of a purely exemplary exemplary processor with illustrative logic units / resources of the processor is illustrated. It is noted that the processor may include or omit any of these functional units, as well as include any other known functional units, logic, or firmware not depicted. As illustrated, core 1401 includes a simplified representative out-of-order (OOO) processor core. However, an in-order processor may be utilized in different embodiments. The OOO core includes a branch target buffer 1420 for predicting branches to be executed / taken and an instruction translation buffer (I-TLB) 1420 for storing address translation entries for instructions.

[0121] The core 1401 further includes a decoding module 1425 coupled to the extraction unit 1420 to decode the extracted elements. The extraction logic includes, in one embodiment, respective sequencers associated with the thread slots 1401a, 1401b. Typically, the core 1401 is associated with a first ISA that defines / specifies instructions executable on the processor 1400. Machine code instructions that are part of the first ISA often include a portion of instructions (called opcodes) that reference / specify the instructions or operations to be performed. The decoding logic 1425 includes circuits that recognize these instructions from their opcodes and pass decoded instructions in the pipeline for processing as defined by the first ISA. For example, as described in more detail below, the decoder 1425 includes, in one embodiment, logic designed or adapted to recognize specific instructions such as transactional instructions. As a result of the decoder 1425 recognition, the architecture or core 1401 takes specific predefined actions to perform tasks associated with the appropriate instructions. It is important to note that any of the tasks, blocks, operations, and methods described herein may be executed in response to a single or multiple instructions; some of which may be new instructions or old instructions. Note that decoder 1426 recognizes the same ISA (or a subset thereof) in one embodiment. Alternatively, in a heterogeneous core environment, decoder 1426 recognizes a second ISA (either a subset of the first ISA or a distinct ISA).

[0122] In one example, allocator and renamer block 1430 includes an allocator that reserves resources, such as a register file that stores instruction processing results. However, threads 1401a and 1401b are potentially capable of out-of-order execution, where allocator and renamer block 1430 also reserves other resources, such as a reorder buffer, to track instruction results. Unit 1430 may also include a register renamer to rename program / instruction reference registers to other registers internal to processor 1400. Reorder / retirement unit 1435 includes components such as the reorder buffer, load buffer, and store buffer mentioned above to support out-of-order execution and subsequent in-order retirement of instructions executed out-of-order.

[0123] In one embodiment, scheduler and execution unit block 1440 includes a scheduler unit to schedule instructions / operations on execution units. For example, floating point instructions are scheduled on execution unit ports with available floating point execution units. Register files associated with execution units are also included to store information instruction processing results. Example execution units include floating point execution units, integer execution units, jump execution units, load execution units, store execution units, and other known execution units.

[0124] A lower level data cache and data translation buffer (D-TLB) 1450 is coupled to the execution unit 1440. The data cache is to store recently used / operated units, such as data operands, which are potentially kept in a memory consistency state. The D-TLB is to store virtual / linear recent translations to physical addresses. As a specific example, the processor may include a page table structure to divide the physical memory into multiple virtual pages.

[0125] Here, cores 1401 and 1402 share access to a higher level or more cache, such as a second level cache associated with on-chip interface 1410. It is noted that higher level or more refers to a cache level that increases or gets further way from the execution unit. In one embodiment, the higher level cache is a last level data cache - the last cache in the memory hierarchy on processor 1400 - such as a second level or third level data cache. However, the higher level cache is not so limited because it can be associated with or include an instruction cache. A trace cache - a type of instruction cache - can instead be coupled behind decoder 1425 to store recently decoded traces. Here, an instruction potentially refers to a macroinstruction (i.e., a general instruction recognized by a decoder), which can be decoded into a number of microinstructions (micro-operations).

[0126] In the depicted configuration, the processor 1400 also includes an on-chip interface module 14140. Historically, a memory controller, described in more detail below, has been included in a computing system external to the processor 1400. In this case, the on-chip interface 1410 is to communicate with devices external to the processor 1400, such as system memory 1475, a chipset (which in some cases includes a memory controller hub or a shared memory controller connected to the memory 1475 and an I / O controller hub to which peripheral devices are connected), a memory controller hub, a north bridge, or other integrated circuits. And in this case, the link 1405 may include any known interconnect, such as a multi-drop bus, a point-to-point interconnect, a serial interconnect, a parallel bus, a consistent (e.g., cache consistent) bus, a layered protocol architecture, a differential bus, a GTL bus, or an SML link.

[0127] Memory 1475 may be dedicated to processor 1400, or shared with other devices in the system. Common examples of types of memory 1475 include DRAM, SRAM, non-volatile memory (NV memory), and other known storage devices. It is noted that device 1480 may include a graphics accelerator, a card or processor coupled to a memory controller hub, data storage coupled to an I / O controller hub, a wireless transceiver, a flash memory device, an audio controller, a network controller, or other known devices.

[0128] However, recently, when more logic and devices are integrated on a single die (such as a SOC), each of these devices can be combined on the processor 1400. For example, in one embodiment, the memory controller hub is on the same package and / or die as the processor 1400. Here, the core portion (core portion) 1400 includes one or more controllers for interfacing with other devices such as memory 1475 or graphics device 1480. Configurations that include controllers and interconnects for docking with such devices are often referred to as core (or non-core) configurations. As an example, the on-chip interface 1410 includes a ring interconnect for on-chip communication and a high-speed serial point-to-point link 1405 for off-chip communication. However, in a SOC environment, even more devices, such as network interfaces, coprocessors, memory 1475, graphics processors 1480, and any other known computer devices / interfaces can be integrated on a single die or integrated circuit to provide a small form factor with high functionality and low power consumption.

[0129] In one embodiment, processor 1400 can execute compiler, optimization and / or converter code 1477 to compile, convert and / or optimize application code 1476 to support the equipment and method described herein, or dock with it. Compiler often includes a program or a group of programs that converts source text / code into target text / code. Usually, compiler compiler / application code is carried out in multiple stages and multiple passes to transform high-level programming language code into low-level machine or assembly language code. However, for simple compilation, single-pass compiler can still be used. Compiler can utilize any known compilation technology and perform any known compiler operation, such as lexical analysis, pre-processing, parsing, semantic analysis, code generation, code conversion and code optimization.

[0130] Larger compilers often include multiple phases, but most often, these phases are included within two general phases: (1) the front end, i.e., generally where syntactic processing, semantic processing, and some transformations / optimizations may occur, and (2) the back end, i.e., generally where analysis, transformations, optimizations, and code generation may occur. Some compilers refer to the middle, which illustrates the blurring of the characterization between the front end and back end of a compiler. As a result, references to inserting, associating, generating, or other operations of a compiler may occur in any of the above-mentioned phases or passes, as well as any other known phases or passes of a compiler. As an illustrative example, a compiler potentially inserts operations, calls, functions, etc., in one or more phases of compilation, such as inserting a call / operation in the front end of compilation, and then transforming the call / operation into lower-level code during the transformation phase. It is noted that during dynamic compilation, compiler code or dynamic optimization code may insert such operations / calls, as well as optimize the code for execution during runtime. As a specific illustrative example, binary code (already compiled code) may be dynamically optimized during runtime. Here, program code may include dynamic optimization code, binary code, or a combination thereof.

[0131] Similar to compilers, translators (such as binary translators) statically or dynamically translate code to optimize and / or convert code. Thus, references to executing code, application code, program code, or other software environments may refer to: (1) dynamically or statically executing a compiler program, code optimizer, or translator to compile program code, maintain software structure, perform other operations, optimize code, or convert code; (2) executing a main program code including operations / calls, such as application code that has been optimized / compiled; (3) executing other program code, such as a library associated with the main program code that maintains software structure, to execute other software-related applications, or optimize code; or (4) a combination thereof.

[0132] While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will recognize numerous modifications and variations therefrom. The appended claims are intended to cover all such modifications and variations that fall within the true spirit and scope of the present invention.

[0133] The design may go through various stages from creation to simulation to manufacturing. Data representing the design may represent the design in a variety of ways. First, as useful in simulation, the hardware may be represented using a hardware description language or another functional description language. In addition, a circuit-level model with logic and / or transistor gates may be generated at a certain stage in the design process. In addition, most designs reach a data level representing the physical placement of various devices in the hardware model at a certain stage. In the case of using conventional semiconductor manufacturing technology, the data representing the hardware model may be data specifying the presence or absence of various features on different mask layers of a mask for producing an integrated circuit. In any representation of the design, the data may be stored in any form of machine-readable medium. Memory or magnetic storage or optical storage, such as a disk, may be a machine-readable medium to store information transmitted via light or electric waves, which is modulated or otherwise generated to transmit such information. When an electric carrier indicating or carrying a code or design is transmitted, a new copy is made to the extent that the duplication, buffering or retransmission of the electric signal is performed. Therefore, a communication provider or a network provider may store products such as information encoded into a carrier at least temporarily on a tangible, machine-readable medium, thereby implementing the technology of an embodiment of the present invention.

[0134] As used herein, a module refers to any combination of hardware, software, and / or firmware. For example, a module includes hardware, such as a microcontroller, associated with a non-transitory medium to store code adapted to be executed by the microcontroller, and therefore, in one embodiment, a reference to a module refers to hardware that is particularly configured to recognize and / or execute code to be retained on a non-transitory medium. In addition, in another embodiment, the use of a module refers to a non-transitory medium including code, which is particularly adapted to be executed by a microcontroller to perform a predetermined operation. And as can be inferred, in yet another embodiment, the term module (in this example) may refer to a combination of a microcontroller and a non-transitory medium. Typically, module boundaries that are described as separate typically change and potentially overlap. For example, a first and a second module may share hardware, software, firmware, or a combination thereof, while potentially retaining some independent hardware, software, or firmware. In one embodiment, the use of the term logic includes hardware, such as transistors, registers, or other hardware, such as programmable logic devices.

[0135] In one embodiment, the use of the phrase "configured to" refers to arranging, putting together, manufacturing, offering for sale, importing and / or designing a device, hardware, logic or element to perform a specified or determined task. In this example, if it is designed, coupled and / or interconnected to perform the specified task, then no device or element that is operating is still "configured to" perform the specified task. As an illustrative example only, a logic gate can provide a 0 or 1 during operation. But a logic gate is "configured to" provide an enable signal to the clock, not including every potential logic gate that can provide a 1 or 0. Instead, a logic gate is a logic gate coupled in a way that a 1 or 0 output is used to enable the clock during operation. Note again that the use of the term "configured to" does not require operation, but instead focuses on the potential state of the device, hardware and / or element, where the potential state of the device, hardware and / or element is designed to perform a specific task when the device, hardware and / or element is operating.

[0136] Additionally, in one embodiment, the use of the phrases "to," "can," and / or "operable to" refers to some devices, logic, hardware, and / or elements designed in such a manner that enables the device, logic, hardware, and / or element to be used in a specified manner. As noted above, in one embodiment, the use of to, can, or operable to, refers to a potential state of a device, logic, hardware, and / or element, wherein the device, logic, hardware, and / or element is not operating but is designed in such a manner that enables the device to be used in a specified manner.

[0137] As used herein, a value includes any known representation of a number, state, logic state, or binary logic state. Typically, the use of a logic level, a logic value, or a logical value is also referred to as 1 and 0, which represent only a binary logic state. For example, 1 refers to a high logic level and 0 refers to a low logic level. In one embodiment, a storage unit, such as a transistor or a flash memory unit, may be able to hold a single logic value or multiple logic values. However, other representations of values ​​in a computer system are used. For example, the decimal number 10 may also be represented as a binary value of 1410 and the letter A in hexadecimal. Therefore, a value includes any representation of information that can be held in a computer system.

[0138] In addition, states may be represented by values ​​or portions of values. For example, a first value, such as a logical 1, may represent a default or initial state, while a second value, such as a logical 0, may represent a non-default state. In addition, in one embodiment, the terms reset and set, refer to default and updated values ​​or states, respectively. For example, a default value potentially includes a high logical value, i.e., reset, while an updated value potentially includes a low logical value, i.e., set. Note that any combination of values ​​may be used to represent any number of states.

[0139] Embodiments of the methods, hardware, software, firmware, or code set forth above may be implemented via instructions or code stored on a machine-accessible, machine-readable, computer-accessible, or computer-readable medium that is executable by a processing element. Non-transitory machine-accessible / readable media include any mechanism that provides (i.e., stores and / or transmits) information in a form that is readable by a machine, such as a computer or electronic system. For example, non-transitory machine-accessible media include random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM); ROM; magnetic or optical storage media; flash memory devices; electrical storage devices; optical storage devices; acoustic storage devices; other forms of storage devices for holding information received from a transient (propagating) signal (e.g., a carrier wave, an infrared signal, a digital signal); etc., which would be distinguished from a non-transitory medium from which information may be received.

[0140] Instructions for programming logic to perform embodiments of the present invention may be stored in a system's memory, such as a DRAM, cache, flash memory, or other storage device. In addition, instructions may be distributed via a network or by means of other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but not limited to, a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible, machine-readable storage device for transmitting information via an electrical, optical, acoustic, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.) through the Internet. Therefore, computer-readable media include any type of tangible machine-readable media that is suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0141] The following examples pertain to embodiments according to the present specification. One or more embodiments may provide an apparatus, system, machine-readable storage, machine-readable medium, method, and hardware-based and / or software-based logic (e.g., implemented in the form of a shared memory controller) for the following: a first interface for receiving a microchip associated with a memory transaction. A particular shared memory controller is used to control access to a portion of a shared memory resource having a plurality of memory elements. The microchip may include a node identifier field for identifying a particular node among a plurality of processor nodes that is the source of the transaction.

[0142] In one example, the microsheet includes at least three slots.

[0143] In one example, at least two of the slots are used to identify addresses of two different transactions, and a third slot of the at least three slots is used to provide bits for extending addresses identified in the at least two of the slots.

[0144] In one example, the flits further include a field for indicating whether the third slot is used to extend addresses identified in the at least two of the slots.

[0145] In one example, when the field indicates that the third slot is not used to extend the range of addresses identified in the at least two of the slots, the flits have a format of a single-node buffer memory link protocol.

[0146] In one example, the specific shared memory controller is further configured to decode the flits and determine a route for the flits based on the decoding.

[0147] In one example, the specific shared memory controller further includes a second interface for routing the flits to another shared memory controller of the plurality of shared memory controllers based on the decoding.

[0148] In one example, the another shared memory controller is implicitly determined based on an address field of the flits.

[0149] In one example, the another shared memory controller is determined based on a shared memory controller number encoded in a destination node field of the flits.

[0150] In one example, the particular shared memory controller is directly connected to a particular subset of the plurality of memory elements corresponding to the portion of the shared memory resource and manages access to the particular subset.

[0151] In one example, the first interface includes a layered protocol stack.

[0152] In one example, the first interface utilizes a shared memory link protocol, and the shared memory link protocol utilizes physical layer logic of a different interconnect protocol.

[0153] In one example, the different interconnect protocol includes a Peripheral Component Interconnect Express (PCIe) based protocol.

[0154] One or more embodiments may provide an apparatus, system, machine-readable storage, machine-readable medium, method, and hardware-based and / or software-based logic (e.g., implemented in the form of a shared memory controller) for receiving a microchip from another first shared memory controller via a shared memory link, wherein the microchip includes a node identifier (ID) field and an address of a specific row of shared memory, and the node ID field identifies the source of the first shared memory controller corresponding to the microchip. In addition, a second shared memory controller is determined based on at least the address field of the microchip, wherein the second shared memory controller is connected to a memory element corresponding to the specific row. The microchip can be forwarded to the second shared memory controller using a shared memory link according to a routing path.

[0155] In one example, a routing path to the second shared memory controller is to be determined.

[0156] In one example, the routing path is one of a plurality of alternate routing paths to the second shared memory controller, and the shared memory controller is operable to determine a particular one of the plurality of alternate routing paths based on a field in the flits.

[0157] In one example, another microchip is generated to correspond to another transaction involving another row of the shared memory, the other microchip including a node ID field and an address of the other row of the shared memory, and the node ID field of the other microchip is used to identify the source of the shared memory controller corresponding to the other transaction.

[0158] One or more embodiments may provide a system comprising: a shared memory including a plurality of memory elements; a shared memory controller for managing access to a subset of the plurality of memory elements; and at least one processor node connected to the shared memory controller via a shared memory link. The processor node is configured to generate a microchip corresponding to a transaction to be related to a specific address of the shared memory and send the microchip to the shared memory controller. The shared memory controller may also identify a specific shared memory controller corresponding to the specific address among a plurality of shared memory controllers and determine a routing path for providing the microchip to the specific shared memory.

[0159] In one example, the shared memory controller is used to deliver the result of the transaction to the processor node.

[0160] In one example, the shared memory controller includes a first shared memory controller, the subset of the plurality of memory elements includes a first subset of the memory elements, the processor node includes a first processor node, and the system further includes: a first device including the first shared memory controller, the first subset of the memory elements, and the one or more nodes; and a second device connected to the first device via a shared memory link, wherein the second device includes a second shared memory controller, a second subset of the memory elements, and a second processor node.

[0161] References throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the various appearances of the phrases "in one embodiment" or "in an embodiment" throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0162] In the foregoing description, detailed descriptions have been given with reference to specific exemplary embodiments. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative sense rather than a restrictive sense. Furthermore, the foregoing use of embodiments and other exemplary language do not necessarily refer to the same embodiment or the same example, but may refer to different and distinct embodiments as well as potentially referring to the same embodiment.

Claims

1. A device for accessing a computer memory, comprising: Network controller and shared memory controller; one or more central processing unit (CPU) nodes coupled to the shared memory controller; An interface supporting multiple layered protocol stacks, the multiple layered protocol stacks including a memory access protocol stack and a peripheral component interconnect express (PCIe) protocol stack, The layered protocol stack has a first layer, a second layer, and a third layer, wherein the third layer includes a physical layer, The physical layer is shared by the memory access protocol stack and the PCIe protocol stack to physically transmit information to an external device, and the physical layer includes: A transmitter transmits data to an external device via a link that aggregates multiple channels, a receiver, receiving data from the external device via the link, Wherein a channel represents a differential signal pair for transmission and a differential signal pair for reception; as well as Separate pin groups are used to implement the memory access protocol stack and the PCIe protocol stack.

2. The apparatus according to claim 1, further comprising: The conversion logic is used to convert the address value referenced by the CPU node in the memory access request into a global address value in the pooled memory domain.

3. The apparatus according to claim 2, further comprising: internal routing logic to: facilitate routing of memory access requests to the local memory element if the translation logic indicates that the shared memory controller has authority to directly access a particular row of pooled memory hosted on the local memory element; as well as Global address routing logic is configured to determine, based on the global address value, that the memory access request is to be routed to a remote shared memory controller coupled to a remote memory element associated with the global address value.

4. The apparatus according to claim 1, further comprising: A multiplexer is coupled to the physical layer to transmit and receive information according to the memory access protocol stack or the PCIe protocol stack.

5. The device according to claim 3, wherein: The first layer includes a transaction layer for assembling and disassembling packets, and the second layer includes a data link layer to provide a reliable mechanism for exchanging the packets over the link.

6. The device according to claim 5, wherein: The data link layer calculates and applies error detection codes and submits the modified packet to the physical layer for transmission over the link.

7. The device according to claim 6, wherein: The error detection code is a cyclic redundancy code CRC.

8. The device according to claim 7, wherein: The CRC comprises a 16-bit value.

9. The device according to claim 8, wherein: The CPU nodes include a first CPU node and a second CPU node, wherein the first CPU node and the second CPU node are operable as independent nodes under the control of a virtual machine monitor.

10. The device according to claim 3, wherein: The first layer includes a coherency protocol to support cache lines of data from the memory elements.

11. The device according to claim 2, wherein: The translation logic includes a translation lookaside buffer.

12. The device according to any one of claims 1 to 11, wherein: The CPU node also includes: a decoder for decoding the fetched instructions; and The scheduler is used to schedule instructions on the execution unit.

13. An integrated circuit die, comprising: multiple controllers, including a network controller and a shared memory controller; a processor coupled to the plurality of controllers; as well as a multi-channel communication interface coupled to the plurality of controllers, the multi-channel communication interface comprising an electrical sub-block to physically transfer information to an off-die device, the electrical sub-block comprising: a transmitter for transmitting information to the off-die device via a first differential signal path; and a receiver for receiving information from the off-die device via a second differential signal path, the transmitter and the receiver corresponding to one of a plurality of channels of the multi-channel communication interface; The multi-channel communication interface supports a first layered protocol stack and a second layered protocol stack, the first layered protocol stack and the second layered protocol stack are overlaid on the electrical sub-block, and the electrical sub-block can be switched to transmit and receive information according to the first layered protocol stack, or to transmit and receive information according to the second layered protocol stack.

14. The integrated circuit die of claim 13, wherein: The first layered protocol stack includes a first link layer implemented in a first link layer circuit, and the second layered protocol stack includes a second link layer different from the first link layer, the second link layer being implemented in a second link layer circuit.

15. The integrated circuit die of claim 13, wherein: The transmitter is used to serialize information for transmission to the off-die device, and the receiver is used to deserialize information received from the off-die device.

16. The integrated circuit die of any one of claims 13-15, wherein: The multi-channel communication interface is a serial point-to-point link interface.

17. The integrated circuit die of claim 14, further comprising a logic PHY circuit coupled to the first link layer circuit and the second link layer circuit to prepare outgoing information for transmission through the electrical sub-block, the outgoing information comprising: Error detection code associated with a microchip.

18. The integrated circuit die of claim 17, wherein: The microchip is a multi-grooved microchip.

19. The integrated circuit die of claim 17, wherein: The error detection code is a CRC value.

20. The integrated circuit die of claim 17, wherein: The error detection code is a 16-bit CRC value.

21. An integrated circuit die as claimed in any one of claims 13 to 15, wherein: The processor includes a first processor core and a second processor core, wherein the first processor core and the second processor core are operable as independent nodes under the control of a virtual machine monitor.

22. An integrated circuit die as claimed in any one of claims 13 to 15, wherein: The first layered protocol stack is an input / output (I / O) protocol stack.

23. An integrated circuit die as claimed in any one of claims 13 to 15, wherein: The first layered protocol stack is a network protocol stack.

24. An integrated circuit die as claimed in any one of claims 13 to 15, wherein: The second layered protocol stack is a memory access protocol stack.

25. The integrated circuit die of claim 24, wherein: The memory access protocol stack is a consistent memory access protocol stack.

26. The integrated circuit die of any one of claims 13 to 15, further comprising: a first set of transmission paths for transmitting serial data according to the first layered protocol stack; and a second set of transmission paths for transmitting serial data according to the second layered protocol stack.

27. The integrated circuit die of claim 26, wherein: The first group of transmission paths and the second group of transmission paths include respective copper wires.

28. An integrated circuit die, comprising: a plurality of controllers including a first shared memory controller SMC to be coupled to a local memory device and at least one remote memory device; a processor coupled to the plurality of controllers; as well as a multi-channel communication interface coupled to the plurality of controllers, the multi-channel communication interface comprising an electrical sub-block to physically transmit and receive information, the electrical sub-block comprising: a transmitter for transmitting information; and a receiver for receiving information, the transmitter and the receiver corresponding to one of a plurality of channels of the multi-channel communication interface; wherein the multi-channel communication interface supports a first layered protocol stack and a second layered protocol stack, the first layered protocol stack and the second layered protocol stack being overlaid on the electrical sub-block, wherein the electrical sub-block is switchable to transmit and receive information according to the first layered protocol stack or to transmit and receive information according to the second layered protocol stack, The second layered protocol stack is a memory access protocol stack, and the memory access protocol stack includes an upper protocol layer to route a memory access request to the remote memory device or the local memory device.

29. The integrated circuit die of claim 28, wherein: The upper protocol layer includes a transaction layer or a link layer of the memory access protocol stack.

30. An integrated circuit die as claimed in any one of claims 28 or 29, wherein: To route the memory access request to the remote memory device, the upper protocol layer is used to route the memory access request to a second die through one or more of the plurality of channels, the second die comprising a second SMC coupled to the remote memory device.

31. The integrated circuit die of any one of claims 28-29, wherein: The first layered protocol stack includes a first link layer implemented in a first link layer circuit, and the second layered protocol stack includes a second link layer different from the first link layer, the second link layer being implemented in a second link layer circuit.

32. The integrated circuit die of any one of claims 28-29, wherein: The transmitter is used to serialize information for transmission to an off-die device, and the receiver is used to deserialize information received from the off-die device.

33. The integrated circuit die of any one of claims 28-29, wherein: The multi-channel communication interface is a serial point-to-point link interface.

34. The integrated circuit die of claim 31 further comprising a logic PHY circuit coupled to the first link layer circuit and the second link layer circuit to prepare outgoing information for transmission through the electrical sub-block, the outgoing information comprising: Error detection code associated with a microchip.

35. The integrated circuit die of claim 34, wherein: The microchip is a multi-grooved microchip.

36. The integrated circuit die of claim 34, wherein: The error detection code is a CRC value.

37. The integrated circuit die of claim 34, wherein: The error detection code is a 16-bit CRC value.

38. The integrated circuit die of any one of claims 28-29, wherein: The processor includes a first processor core and a second processor core, wherein the first processor core and the second processor core are operable as independent nodes under the control of a virtual machine monitor.

39. The integrated circuit die of any one of claims 28-29, wherein: The first layered protocol stack is an input / output (I / O) protocol stack.

40. The integrated circuit die of any one of claims 28-29, wherein: The first layered protocol stack is a network protocol stack.

41. The integrated circuit die of any one of claims 28-29, wherein: The memory access protocol stack is a consistent memory access protocol stack.

42. The integrated circuit die of any of claims 28-29, further comprising: a first set of transmission paths for transmitting serial data according to the first layered protocol stack; and a second set of transmission paths for transmitting serial data according to the second layered protocol stack.

43. The integrated circuit die of claim 42, wherein: The first group of transmission paths and the second group of transmission paths include respective copper wires.

Images