Computer Systems

Abstract

The memory attach / routing chip includes a single die having a set of external ports, at least one memory attach interface including a memory controller and attaching to an external memory, and a fabric core in which routing logic is implemented, the routing logic (i) receiving a first packet of a first type from a first external port of the set of external ports, the first packet being a memory access packet including a memory address within a memory address range associated with the memory attach / routing chip, detecting the memory address and routing the first packet to the memory attach interface, and (ii) receiving a second packet of a second type, the second packet being a processor-to-processor packet including a destination identifier that identifies a processing chip external to the memory attach / routing chip, and routing the second packet to a second external port of the set of external ports selected based on the destination identifier.

Description

[Technical field]

[0001] The present disclosure relates to computer systems, and particularly but not exclusively to computers that include one or more processing chips coupled to computer memory. [Background technology]

[0002] The demand for high performance computing is constantly increasing. In particular, efforts are being made to satisfy the demands of artificial intelligence / machine learning models that impose challenging requirements on computing resources. It is known to address some of the requirements by interconnecting multiple processing chips in a cluster, where the multiple processing chips are configured to work in a cooperative manner to satisfy the demands of processing power needed to process large AI / ML models.

[0003] Another requirement imposed by high performance computing is the ability to access high capacity memory. Attempts have been made to connect external memories to the processing nodes in the cluster to increase the capacity of memory. Such external memories may be connected by memory buses that provide an access path between the external memories and the processing nodes. These memory buses may take the form of parallel or serial links. For example, dynamic random access memories (DRAMs) may be mounted on dual in-line memory modules (DIMMs) on a server rack. These can provide scalable memory capacities on the order of terabytes. Such DIMMs may be mounted vertically within the server rack, and many may be stacked together to provide the required memory capacity for the computer. Summary of the Invention [Means for solving the problem]

[0004] The inventors seek to address the connectivity problem by providing a memory attachment and routing chip that not only allows access to high capacity memory, but also allows processing chips to communicate with each other to enhance processing power for specific tasks.

[0005] According to one aspect of the disclosure, a memory attach / routing chip is provided. The memory attach / routing chip includes a single die having a set of external ports, at least one memory attach interface including a memory controller and configured to attach to an external memory, and a fabric core in which routing logic is implemented. The routing logic (i) receives a first packet of a first type from a first external port of the set of external ports, the first packet being a memory access packet including a memory address within a memory address range associated with the memory attach / routing chip, detects the memory address, and routes the first packet to the memory attach interface. The routing logic further (ii) receives a second packet of a second type, the second packet being a processor-to-processor packet including a destination identifier that identifies a processing chip external to the memory attach / routing chip, and routes the second packet to a second external port selected from the set of external ports based on the destination identifier.

[0006] The routing logic may be configured to examine an incoming packet to determine whether the incoming packet is a memory access packet for routing to a memory controller or an inter-processor packet intended for one of the external ports, and to route the incoming packet based on this determination.

[0007] Each external port may be connected to a link and includes at least one link controller formed as circuitry within the die for controlling the transmission of messages on / off the memory attachment / routing chip via the link.

[0008] In some embodiments, the link is a SERDES link, and the link controller includes circuitry configured to control transmission and reception of packets on the SERDES link, which may include digital and / or analog circuitry.

[0009] The first group of external ports can be connected to a serial link configured to be attached to a processor chip, and the routing logic is configured to detect that the second packet includes a processor identifier to identify one of the first group of external ports.

[0010] The memory attachment / routing chips may be connected within a first computer cluster, and at least one of the external ports is attached to a cluster connection link configured to connect to another memory attachment / routing chip in a second computer cluster.

[0011] In some embodiments, at least one of the external ports is connected to a system facing link configured to connect to a switching fabric, and the routing logic is configured to detect a domain identifier in the first packet or the second packet and route the packet to the system facing link.

[0012] The set of external ports may include host ports that are connected to links configured to connect to host computers.

[0013] A memory attachment / routing chip may include multiple memory controllers, and each memory controller is attached to a dynamic random access memory (DRAM) via a DRAM interface component formed within the die.

[0014] Each DRAM interface may be configured according to JEDEC standards.

[0015] In some embodiments, the die is rectangular and the set of external ports is located along one long side of the die.

[0016] The memory attachment interface may include memory attachment connections for connecting the at least one memory controller to an external memory, the memory attachment connections also being disposed along another long side of the die.

[0017] The die may have an aspect ratio of greater than 3:1.

[0018] The die can include a plurality of bumps for flip-chip attachment of the die to the substrate, and the bumps include a first group in electrical communication with the external port and a second group in electrical communication with the memory attachment interface.

[0019] A third group of bumps may be provided that is in electrical communication with the memory controller power supply.

[0020] A fourth group of bumps in electrical communication with the chip power supply may be provided.

[0021] The routing logic may be configured to receive a third type packet including a memory address within a second range of memory addresses outside the range associated with the memory attach / routing chip, determine a second memory attach / routing chip associated with the second range of addresses, and route the packet to the second memory attach / routing chip via one of the external ports.

[0022] A memory attachment and routing chip may or may not include a processor. A memory attachment and routing chip may not include processing logic for executing instructions.

[0023] The first packet and / or the second packet may remain unaltered by the memory attached / routing chip. In other words, the routing logic may be configured to examine the incoming packet without modifying its contents. The memory attached / routing chip may otherwise be configured to route the incoming packet without processing it. Thus, an incoming packet received at the memory attached / routing chip is output from the memory attached chip via the memory attached interface or via a second one of the external ports in substantially the same form as it was received.

[0024] The inventors have recognized shortcomings associated with the current nature of cluster connections. Silicon chips contain circuitry that is laid out in two dimensions on the die surface and is typically divided into "cores" (such as processor core 2) surrounded by a periphery or "beachfront" 7 (see FIG. 1). The beachfront is used for input / output (IO) circuitry that is placed on the edge of the chip to facilitate the division of signals to packaging pins. For example, the beachfront 7 is used to accommodate the inter-processor links 6a and 6b and the processor-memory links 8a...8d to external memory shown in FIGS. 2 and 3.

[0025] The area of ​​the beachfront depends on the type and bandwidth of the I / O requirements. High performance computing chips often use close to the maximum manufacturable die size ("full reticle") of about 25.5 x 32.5 mm, and require a beachfront depth of about 2 mm on each of the four sides of the die. With current lithography technology, a full reticle die provides a die core of about 21.5 x about 28.5 mm, which is about 74% of the total die area. The computing resources of the die are constrained by the core percentage, which has led the inventors to realize that the cost of the beachfront area is prohibitive. Figure 1 shows an example of a die with a processor core 2 and a full perimeter beachfront (labeled 7) (on all four sides).

[0026] According to an aspect of the present disclosure, a computing device is provided, the computing device including a first class of first and second computing devices including a first external port and a second external port, at least one memory controller configured to attach to an external memory, and routing logic configurable to route data from the first external port to one of the at least one memory controller and the second external port; The present invention also includes a second class of first and second computing devices, where the first computing device of the second class is connected to a first external port via a first link and a second link, and the second computing device of the second class is connected to a second external port via a third link and a fourth link, each of the first and second computing devices of the second class being configured to execute one or more computer programs and including processing circuitry connected to the first and second links or the third and fourth links for transmitting and receiving messages via the first and second links or the third and fourth links.

[0027] In some embodiments, the first computing device of the second class may be configured to send a message to a first external port of the first computing device or second computing device of the first class via the first link or the second link, and routing logic on the first computing device or second computing device of the first class is configured to route an incoming message to a second external port of the first computing device or second computing device of the first class for sending the message to the second computing device of the second class.

[0028] In some embodiments, each of the first link, the second link, the third link, and the fourth link is a fixed "point-to-point" link with no intermediate switches. In such embodiments, the first link, the second link, the third link, and the fourth link may be serial links.

[0029] In some embodiments, the first computing device of the second class may be configured to send the memory access message to the first computing device or the second computing device of the first class, and the routing logic of the first computing device or the second computing device of the first class is configured to route the memory access message to at least one memory controller for accessing the external memory.

[0030] In some embodiments, the memory access message is a memory write.

[0031] In some embodiments, the memory access message is a memory read.

[0032] In some embodiments, each computing device of the first class includes a single chip having a fabric core in which routing logic is implemented and a port connection region extending along one long side of the chip along which the external ports are located.

[0033] In some embodiments, each computing device of the second class includes a single chip having a processor core on which the processing circuitry is implemented and a processor port connection area located along at least one edge of the chip on which the processor ports are located and connected to the first link and the second link or the third link and the fourth link, and the chip includes another processor port connection area along an opposite edge of the chip, the another processor port connection area including another processor port for connection to an additional link connected to an additional computing device of the first class.

[0034] In some embodiments, the processor core extends substantially to the other side of the processor chip.

[0035] In some embodiments, the other side of the chip includes a high-bandwidth memory connector configured to connect the processor chip to at least one high-bandwidth memory device.

[0036] In some embodiments, each computing device of the first class includes one or more memory attachment interfaces for attaching the at least one memory controller to external memory.

[0037] In some embodiments, a computer includes a cluster of n computing devices of a first class and N computing devices of a second class, where n is 2 or more and N is greater than 2, and where each computing device of the second class is connected to the n computing devices of the first class via a respective fixed link, and where each computing device of the first class is connected to the N computing devices of the second class via a respective fixed link.

[0038] In some embodiments, a computer includes a cluster of n computing devices of a first class and N computing devices of a second class, where n is greater than 2 and N is equal to or greater than 2, and where each computing device of the second class is connected to the n computing devices of the first class via a respective fixed link, and each computing device of the first class is connected to the N computing devices of the second class via a respective fixed link.

[0039] In some embodiments, a computer includes a cluster of n computing devices of a first class and N computing devices of a second class, where n is greater than 2 and N is greater than 2, each computing device of the second class is connected to the n computing devices of the first class via a respective fixed link, and each computing device of the first class is connected to the N computing devices of the second class via a respective fixed link.

[0040] In some embodiments of the computer, n=N.

[0041] In some embodiments, n is greater than N.

[0042] In some embodiments, there are no direct connections between the first class computing devices or the second class computing devices in the cluster.

[0043] In some embodiments, the computer includes a first class of n computing devices and a second cluster of a second class of N computing devices, where n is greater than 2 and N is greater than 2, each computing device of the second class is connected to the n computing devices of the first class via a respective fixed link and each computing device of the first class is connected to the N computing devices of the second class via a respective fixed link, and the computer includes at least one cluster connecting link connected between the first class of computing devices in the first cluster and the first class of computing devices in the second cluster.

[0044] In some embodiments, the computer may include a switch fabric, and at least one of the first class of devices in the first cluster or the second cluster includes a system connection configured to connect the first class of devices to the switch fabric.

[0045] In some embodiments, each device in the first class has a system connection configured to connect the device to the switch fabric.

[0046] In some embodiments, each device in the first class includes a host connector configured to connect the device to a host.

[0047] In some embodiments of the present invention, any processor chip may access any memory attached to any of the fabric chips in the computer cluster. Memory access may be via high speed serial links. Additionally, any processor may exchange packets with any other processor in the computer through the routing logic of the fabric chips.

[0048] In some aspects of the present invention, the inventors have enabled clustering of processing chips in multiple tiers.

[0049] In some aspects of the invention, each processing chip itself has an improved processor core area of ​​a substrate of a particular size.

[0050] According to another aspect, a method for routing packets in a computer system including at least one cluster of processor chips is provided, the method comprising: Transmitting a packet including destination information from a designated external port connected to a memory attachment / routing chip of a cluster of a first processor chip; determining, in routing logic of the memory-attached / routing chip, from the destination information whether the packet is destined for (i) a memory attached to the memory-attached / routing chip, or (ii) a component attached to an external port of the memory-attached / routing chip; Based on this determination, in case (i), routing the packet to a memory controller of the memory attach / routing chip, and in case (ii), routing the packet to an external port of the memory attach / routing chip.

[0051] Another requirement imposed on high performance computing is the ability to have high bandwidth access to high capacity memory. So-called high bandwidth memory (HBM) is currently implemented by providing the memory within the physical structure of the processing node itself. That is, the memory is provided in close proximity to the processing chips that are implemented on a silicon substrate within a package forming the processing node. In practice, the HBM is pushed up against the processing chips on the silicon substrate so as to be as physically close as possible to the processing chips that provide the processing function. Although high bandwidth is thus achieved, there are limitations on the memory capacity based on the physical size of the memory that can be accommodated within this type of structure. Furthermore, such HBM is expensive to manufacture.

[0052] In the field of Artificial Intelligence (AI) and Machine Learning (ML), mathematical models can become very large and require very high amounts of memory to accommodate them. As model size increases, the cost of providing HBM also increases.

[0053] Currently, the lack of availability of high capacity, high bandwidth memory imposes constraints on the size and nature of models that can be used in machine learning / artificial intelligence computers. In particular, the knowledge capacity of a model is a function of the amount of memory that is reasonably accessible. In some embodiments of the present invention, portions of the beachfront are no longer used for connection to external memory and may be made available to the HBM.

[0054] For a better understanding of the present invention and to show how the same may be put into practice, reference will now be made, by way of example, to the accompanying drawings in which: [Brief description of the drawings]

[0055] [Figure 1] FIG. 1 is a schematic block diagram of a chip connected to a memory. [Diagram 2]FIG. 1 is a schematic diagram of multiple interconnected chips. [Diagram 3] FIG. 1 is a block diagram of multiple processor chips connected using a switch core. [Figure 4] 1 is a schematic block diagram of a processor chip having a reduced beachfront. [Diagram 5] 1 is a schematic block diagram of a computer including interconnected processor chips and fabric chips. [Figure 6] 1 is a schematic block diagram of a computer including interconnected processor chips and fabric chips having a higher ratio of fabric chips to processor chips. [Figure 7] 1 is a schematic block diagram of a computer including interconnected processor chips and fabric chips, each processor chip associated with high bandwidth memory. [Figure 8] 1 is a schematic block diagram of a computer including a set of interconnected clusters. [Figure 9] FIG. 2 is a schematic block diagram of a fabric chip. [Figure 10] FIG. 1 is a schematic diagram of an example processor chip. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0056] There are various known methods for interconnecting processing chips with one another to form clusters of processing chips.

[0057] Figure 1 shows an example of a processor chip intended to be connected in a processing cluster. The processing chip 1 comprises a processor core 2 (indicated by cross-hatching) implemented in a silicon die 4. It is convenient to distinguish between a beachfront area, where external links are provided, and a core area of ​​the processing circuitry of the processor core. The beachfront area comprises inter-processor links 6a, 6b and processor-memory links 8a, 8b, 8c, 8d, which are shown in Figure 1 connected to respective DRAMs 10a, 10b, 10c, 10d.

[0058] FIG. 2 shows a cluster of four processing units of the type shown in FIG. 1 with all-to-all direct connections of inter-processor links. Each processor core 2a, 2b, 2c, 2d is shown with three external connectors 5 along the top edge and three external connectors 5' along the bottom edge. In the cluster of FIG. 2, each processor core is connected to the other processor cores by two external connection links attached to the external connectors in the manner shown. See, for example, links L and L' connecting processor core 2a to processor core 2d. Note that there is still a need for a dedicated processor chip-memory bus for attaching DRAM.

[0059] This is just one example of inter-processor connections within a cluster.

[0060] An alternative way of connecting processor chips together in a cluster is to use a switch fabric. Figure 3 is a schematic diagram showing four processor cores 2a, 2b, 2c, 2d connected to each of two switch cores 12a, 12b. Each switch core can be made to route traffic between specific processor cores under program control. In this arrangement, each processor has access to externally connected DRAM.

[0061] In the above examples, each processing chip had access to memory. In some previous examples, the memory could be externally attached memory connected to each processor core and / or high bandwidth memory (HBM) in a cluster connected within the processor package. In either case, the memory attachment uses the "beach front" of the die.

[0062] In some embodiments of the present disclosure, a computer includes a plurality of processor chips and fabric chips arranged in a cluster. The fabric chips act as memory attachment / routing chips. Within the cluster, each processor chip is connected to all of the fabric chips, and each fabric chip is connected to all of the processor chips by all-to-all bipartite connected. There are no direct connections between the fabric chips in the cluster. Furthermore, there are no direct connections between the processor chips. Each fabric chip has routing logic configured to route incoming packets from one processor chip to another processor chip connected to the fabric chip. Furthermore, each fabric chip has a means for attaching to an external memory. The routing logic can route packets between a processor connected to the fabric chip and a memory attached to the fabric chip. The fabric chip includes a memory controller that performs memory control functions to govern memory access to the memory attached to the fabric chip.

[0063] In some embodiments described further herein, clusters of processing chips and fabric chips can be interconnected to form larger computer systems. Each processor chip in a cluster can access any of the memory attached to any of the fabric chips in the cluster. This significantly enhances the memory capacity made available to any particular processor chip.

[0064] The connection configurations described herein have the additional advantage that in some embodiments it is not necessary to use all of the edges of the processor die to surface external connections.

[0065] The inventors have recognized that it is advantageous to limit the beachfront required for connectivity to fewer than all four sides of the die, freeing up more silicon for fabrication of processing "cores." For example, if only the short sides of a full reticle die are used for I / O, the area available for processor cores on the chip increases to about 88% of the total die area, which is about 19% more than if all four sides were used. Figure 4 shows an example of a chip 1' in which the long sides do not accommodate a beachfront, but the top and bottom sides each have a beachfront 7a, 7b.

[0066] Prior art processing cluster connections require the involvement of all peripheral beachfronts (as shown in Figure 1). The implementation of the present connection configuration described herein allows for the use of processor dies that have beachfronts only on the top and bottom edges and no beachfronts on the long edges (as shown in Figure 4).

[0067] According to the described example of the invention, multiple processors are connected in a cluster by using one or more "fabric chips". Each fabric chip provides access to external memory (e.g. DRAM) and also provides routing for inter-processor traffic. Please refer to FIG. 5. FIG. 5 shows four processor chips 20a, 20b, 20c, 20d. Each processor chip includes a processor core area 22a, 22b, 22c, 22d that extends to the long side of the chip. Each processor chip has an upper beachfront area 30a and a lower beachfront area 30b (shown only for chip 20a). The upper beachfront area 30a has a set of external port connections C1, C2, C3, C4 (labeled only on processor chip 20a). It will become apparent that each processor chip also has four external port connections on the upper beachfront area. The lower beachfront area of ​​each processor chip has four external port connections as well (labeled C5, C6, C7, C8). The lower set of external port connections are labeled only on processor chip 20a, it being apparent that other processor chips will have similar sets of external port connections on their lower beachfront areas.

[0068] The cluster of FIG. 5 further includes eight "fabric chips." Each fabric chip includes a fabric core 40a, 40b, . . . 40h. Each fabric chip has a lower beachfront area 44a, . . . 44h with a set of external ports. These external ports are provided within the port connections labeled only on the fabric chip 40a as FC1, FC2, FC3, FC4. It is clear that each fabric chip has a corresponding set of external ports on each lower beachfront area. The upper beachfront area of ​​each fabric chip includes one or more memory attachment interfaces that allow the fabric chip to connect to one or more memories (represented in FIG. 5 as DRAMs 10a, 10b, 10c, 10d, . . . 10p, respectively). For example, the fabric core 40a shown in FIG. 5 is connected to two DRAMs 10a, 10b by suitable memory attachment interfaces provided on the upper beachfront 46a of the fabric chip. Other high capacity memories (e.g., Double Data Rate DRAM (DDR) and others specified thereafter (Low Power DDR (LPDDR) etc.)) may be connected. The high bandwidth connections between the processor chips and the fabric chips in the cluster are "all-to-all bipartite". This means that each processor chip is connected to every fabric chip and each fabric chip is connected to every processor chip. The connections are via links such as L1 between the processor port in a port connection such as C1 and the fabric chip port in a port connection such as FC1. However, note that in the example shown, there are no direct high bandwidth connections between the processor chips or between the fabric chips in the cluster. Furthermore, in the example shown, there is no external memory connected directly to each processor (although there may be high bandwidth memory in the chip package, see below). Each fabric chip provides a routing function that provides a path between every pair of processors and between each processor and the memory attached to the fabric chip.

[0069] Furthermore, the links may be manifested in any suitable manner. Note that the links are fixed links (i.e., providing point-to-point connections). Each link may be connected or reconnected to various ports to set up the computer configuration. Once the computer configuration is set up or operational, the links are not multiplexable and do not fan-in or fan-out. That is, there are no intermediate switches, instead ports on the processor are directly connected to end ports on the fabric chips. Any packet sent on the link will be received at a port on the other end of the fixed link. The links are advantageously bidirectional and preferably (but not required to be) capable of operating in both directions at once. One particular category of communication links is the SERDES link, which has power requirements that are independent of the amount of data carried on the link or the time spent carrying that data. SERDES is an acronym for Serializer / DeSerializer, as such links are known. For example, twisted pair wiring may be used to implement a SERDES link. To transmit a signal on the wires of such a link, power is required to be applied to the wires to change the voltage to generate the signal. SERDES links are characterized in that there is a fixed power for bandwidth capacity on the SERDES link, whether it is used or not. This is due to the need to provide clock information on the link by constantly switching the current or voltage state of the wires even when no data is being transmitted. As is known, data is transmitted by holding the state of the wires to indicate a logic "0" or a logic "1". SERDES links are implemented at each end by circuitry that connects the link layer device to a physical link, such as a copper wire. This circuitry is sometimes called a PHY (physical layer). In this example, packets are transmitted on the link by using layers 1 and 2 of the Ethernet protocol. However, it will be understood that any data transmission protocol may be used.

[0070] The computer described herein has several advantages.

[0071] There is no longer a need to dedicate a fixed proportion of the processor beachfront (and therefore IO bandwidth) to a fixed amount of memory or inter-processor connectivity: all processor IO bandwidth passes through the fabric chip where it can be used on demand for either purpose (memory or inter-processor).

[0072] Under some popular models of multiprocessor computing, such as bulk synchronous parallel (BSP), the peak DRAM bandwidth and peak inter-processor bandwidth usage may not be simultaneous. Thus, the total bandwidth requirement can be satisfied with a smaller processor beachfront, providing a larger core area to the processor chip. BSP itself is known in the art. According to BSP, each processing node performs a computation phase and a switching phase (sometimes called a communication or message passing phase) in alternating cycles. The computation phase and the switching phase are performed by the processing chips that execute instructions. During the computation phase, each processing unit executes one or more computation tasks locally, but does not communicate any results of these computations to other processing chips in the cluster. During the switching phase, each processing chip is allowed to exchange one or more results of processing from a previous computation phase with one or more of the other processing chips in the cluster. It should be noted that different processing chips may be assigned to different groups for synchronization purposes. According to BSP principles, a barrier synchronization is performed at the transition from the computation phase to the exchange phase, or at the transition from the exchange phase to the computation phase, or at both times. That is, it goes without saying that either all processing chips are required to complete their respective computation phases before any in the group is allowed to proceed to the next exchange phase, or all processing chips in the group are required to complete their respective exchange phases before any processing chip in the group is allowed to proceed to the next computation phase, or both of these conditions are enforced. This sequence of exchange and computation phases is repeated over multiple cycles. In BSP terminology, each repeated cycle of exchange and computation phases can be called a "superstep".

[0073] This has the practical effect that there are some situations in which there is no simultaneous use of all links required to access memory (for purposes of completing the computation phase) and links used to exchange data between processing chips in the switching phase. As a result, there is maximal efficient use of fixed links without compromising memory access times or inter-processor switching delays. It will nevertheless be understood that the embodiments described herein have applications other than when used with BSP or other similar synchronous protocols.

[0074] It is possible that links can be dynamically deactivated so as to effectively not consume any power when not in use. However, the activation times and non-deterministic nature of machine learning applications usually make dynamic activation during program execution problematic. As a result, the inventors have determined that it may be better to take advantage of the fact that link power consumption is substantially constant for any particular configuration, and that the best optimization is to maximize the use of physical links as much as possible by maintaining concurrent inter-processor and processor-memory activity.

[0075] All of the memory in a cluster is accessible to each processor, but only indirectly through another processor. This shared memory arrangement benefits software efficiency.

[0076] In the example shown in FIG. 5, there are two "ranks" of fabric chips, one each attached to a corresponding top and bottom side of the processor chip. The top rank includes fabric cores 40a...40d connected to each processor core by corresponding links. For example, processor core 20a is connected to fabric core 40a by link L1, to fabric core 40b by link L2, to fabric core 40c by link L3, and to fabric core 40d by link L4. The bottom rank includes fabric cores 40e...40h. Fabric core 40a is also connected to each of processor cores 20a...20d by corresponding links (shown in the figure but not labeled for clarity reasons). There is no use of the beachfront longitudinal processing chip edges.

[0077] However, there are various design choices within the overall concept: for example, the long edge of the processor could be used to provide more bandwidth to the fabric chip, and all of the links emerging from the beachfront of the processor chip could be passed to a single rank or three ranks, etc. of the fabric chip.

[0078] The number of fabric chips in each rank may differ from the number of processor chips. What remains important to realizing the advantages of the present invention is that an all-to-all bipartite connection between the processing chips and the fabric chips is maintained, with routing functions and external memory access provided by the fabric chips.

[0079] FIG. 6 shows a specific example where four processing chips are connected to an upper rank of eight fabric chips and a lower rank of eight fabric chips. Each processing chip is connected to 16 fabric chips. Take processor chip 20a for example. It has eight upper link connectors C1, C1', C2, C2', C3, C3', C4, C4', each connected to a link connector on a fabric core (two of which are labeled 40a, 40a' in FIG. 6). The processor chip has eight lower link connectors C5, C5', C6, C6', C7, C7', C8, C8', each connected to a respective link connector on each fabric chip of the eight fabric chips in the lower rank. Each fabric chip is connected to four processor chips.

[0080] It should be noted that the use of external connectors to provide all-to-all two-part connectivity within a cluster according to examples of the present invention does not preclude the presence of other I / O ports on the processor chips or fabric chips. For example, some of the processor chips or fabric chips within a cluster may include I / O ports that allow connectivity between multiple clusters, to host devices, etc. In one embodiment described with reference to Figures 8 and 9, the fabric chips provide this additional connectivity.

[0081] Furthermore, it should be noted that additional memory can be directly attached to the processor chip (e.g., along a long edge). That is, additional high bandwidth memory (HBM) can be provided in close proximity to the processing chips implemented on a silicon substrate in the package forming the processing node. In practice, the HBM is pushed up against the processing chips on the silicon substrate so as to be as physically close as possible to the processing chips providing the processing function. For example, the high bandwidth memory (HBM) can be attached to the processor chips while the high capacity memory can be attached to the fabric chips, thus combining the advantages of both memory types in the cluster. FIG. 7 shows an embodiment in which a high bandwidth memory (HBM) module 26 is attached to the east and west edges of each processor chip 20'a, 20'b, 20'c, 20'd. In other respects, the computer shown in FIG. 7 has the same connections as those described in FIG. 5. The attachment of the HBM 26 can be by a short parallel connection of a memory bus formed in the substrate or by using a silicon bridge in the package substrate.

[0082] In the example computer described herein, the processor chips 20 are not intended to be deployed on a stand-alone basis. Instead, their deployment is in a computer cluster where the processor chips are supported by one or more fabric chips 40. The processor chips 20 connect to each other via the fabric chips 40, allowing the use of all of the processor chip links L1, L2, etc. for simultaneous use as processor-to-processor links and memory access links. In this way, the computer provides a high-capacity, high-speed memory system compared to existing computer systems. In current computer systems, it becomes increasingly expensive to provide high-capacity, high-bandwidth memory. Furthermore, there are still limitations on the processing power that can be obtained while delivering high-bandwidth memory access and high-capacity memory. The present computer allows those limitations to be exceeded.

[0083] By providing the routing logic on the fabric chip, the processor chip does not need to have routing logic for external routing functions. This not only allows silicon area to be freed up to maximize processor chip I / O bandwidth, but also maximizes the area available for processing circuitry within the processor core.

[0084] Placing link ports along the North and South edges frees up the East / West edges, either allowing processor cores to extend into the East / West edges, thereby maximizing processing power, or allowing the East / West edges to be kept free for high bandwidth memory integration.

[0085] A computer may be operated in a variety of topologies. In one example, a group of four processor chips and eight fabric chips (e.g., as shown in FIG. 5) may constitute a cluster. Within a cluster, each group of four fabric chips connected to one of the processor chip edges is referred to herein as a rank. The cluster of FIG. 5 includes two ranks.

[0086] A pod may contain multiple clusters. Clusters may be interconnected within a pod using processor-to-processor links on the fabric chip. Pods may be interconnected using pod-to-pod links on the fabric chip. These are shown in more detail in FIG. 9, which shows a fabric chip.

[0087] FIG. 8 is a schematic diagram of a system topology and hierarchy according to one embodiment. FIG. 8 shows multiple pods P1, P2, P3, Pn (labeled POD16). In the example of FIG. 8, n=8, but it is easily understood that various numbers of pods can be connected into a computer system by using the techniques described herein. One of the pods, P1, will be described in detail. Pod P1 includes four clusters Q1, Q2, Q3, Q4. In the example of FIG. 8, each cluster includes four processor chips 20a, 20b, 20c, 20d that share 32 fabric chips. Only one fabric chip 40 (Q4) is labeled in FIG. 8 for clarity reasons. That is, Q4 indicates that the fabric chip 40 (Q4) is in cluster Q4. As shown in FIG. 8, within each cluster Q1, Q2, Q3, Q4, four processor chips 20a, 20b, 20c, 20d are connected to 32 fabric chips in an all-to-all bipartite arrangement. That is, as described above, each fabric chip in the cluster is connected to all four processor chips, and each processor chip is connected to all 32 fabric chips. Each processor chip has 32 port connections C1, C2, ... C32 (16 on the top edge and 16 on the bottom edge). As shown in FIG. 9, in some embodiments, each port connection provides three bidirectional serial links, making a total of 96 processor links. Each set of 12 processor external links (out of the 96 links) connects to four of the fabric chips in the respective set (three processor links for each fabric chip port connection FC). Thus, eight sets of 12 links connect to eight sets of four fabric chips in the cluster. Four clusters, Q1, Q2, Q3, and Q4, are grouped to form a pod by using the pod-to-pod links of the 32 fabric chips in each cluster. Each cluster exports three bundles of 32 links each, with each bundle connecting to each of the other three clusters. The pod-to-pod links of a bundle between two clusters includes one link between each of the 32 corresponding peer fabric chips in the two clusters.Some of the pod-to-pod links may be connected to third party Ethernet switches.

[0088] FIG. 9 is a schematic block diagram of the components on the fabric chip 40. As shown in FIG. 9, routing logic 46 is connected to the DDR interface block 48 for transferring data packets between the DDR interface block 48 and other ports. Routing logic 46 is further attached to each processor-facing link port. Each port includes an Ethernet port controller EPC. Routing logic is attached to the Ethernet port controllers of the pod-facing ports and the Ethernet port controllers of the system-facing links. Routing logic 46 is further attached to a PCI complex for interfacing with a host system. PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high-speed computers.

[0089] FIG. 9 shows an example of a fabric chip that allows computers to be built by connecting computer clusters together in a hierarchical manner, in addition to allowing inter-processor and processor-to-memory communication. First, the parts of the fabric chip used to implement inter-processor and processor-to-memory communication are described. Each fabric core port connection includes three serial links. Each serial link includes a port with an Ethernet port controller (EPC). As previously mentioned, these links can be SERDES links (e.g., twisted pairs that allow serial packet communication).

[0090] For reasons of clarity, not all of the components in FIG. 9 are indicated by associated references. Each of the fabric core connections FC1, FC2, FC3, and FC4 has a configuration as just described herein with reference to the fabric core port connection FC2 that connects to a second processor (e.g., processor 20b in FIG. 6). The fabric connection FC2 includes three links L2a, L2b, L2c, each including an Ethernet port controller EPC2a, EPC2b, EPC2c, respectively. Note that in other embodiments, a single physical link may be provided, or a different number of physical links may be provided within each fabric chip connection FC. Note thus that the link labeled L2 in the previous figures may include three individual serial links (L2a, L2a, L2c, etc.). The routing logic 46 in the fabric chip 40 may be implemented as a ring router, as a crossbar router, or in any other manner. The fabric chip is further connected to external memory (DRAMs 10A, 10B, etc.) (not shown in FIG. 9). While two DRAMs have been shown in the previous figures, in the embodiment of FIG. 9, the fabric chip is connected to four DRAMs. To make this connection, the fabric chip includes four DRAM interface blocks DIB1, DIB2, DIB3, DIB4, each associated with four DDR sub connection layers: DDR sub1, DDR sub2, DDR sub3, DDR sub4. Each DDR interface block DIB 48 incorporates a memory controller that manages access to the memory attached to the block. Although one memory attachment interface 44 is shown in FIG. 9, it will be understood that each DDR sub layer has a respective memory attachment interface for attaching to an external DRAM. The routing logic 46 is configured to route memory access packets received from an attached processor core to a destination one of the data interface blocks DIB1-DIB4.The routing logic 46 is further configured to route packets from one attached processor chip to another attached processor chip via the respective fabric chip port. In some embodiments, the routing logic prevents memory packets (such as memory access response packets) from being routed from one memory attached interface to another. Memory response packets in such embodiments may be routed to the processor chip only via the correct port attached to the routing logic 46. For example, an incoming packet on link L2a of fabric core port connection FC2 will be routed to a destination port connected to the routing logic 46 based on the routing information in the packet. For example, if the packet is intended to be routed to processor 20c, the routing logic 46 will identify processor 20c from the routing information in the packet and cause the packet to exit through an Ethernet port controller onto a link attached to processor 20c.

[0091] If the packet is a memory access packet, the routing logic routes the packet to the appropriate DDR interface block based on the memory address in the packet. Note that in this embodiment, each DDR interface block DIB1...DIB4 includes four memory access channels. It will be appreciated that any number of memory access channels may be provided by each interface block DIB1...DIB4. The memory access channels are managed by a memory controller in each data interface block DIB1...DIB4.

[0092] As explained above, in the example shown in FIG. 9, the fabric chip 40 has additional components that allow a computer to be formed of interconnected clusters. To this end, the fabric chip includes a pod-facing port connection PL. The pod-facing port connection PL includes three ports, each including an Ethernet port controller Pa, Pb, Pc, connected to a link. The routing logic detects a packet whose packet information indicates that the packet should not be routed to a processor in this cluster, but instead to a processor of another cluster, and routes the packet to one of the pod-facing ports. Note that the pod-facing port connection PL may send a packet to a corresponding pod-facing port in a fabric chip on another cluster, or may receive a packet from a corresponding pod-facing port on a fabric chip on another cluster.

[0093] The fabric chip of FIG. 9 also allows packets to be routed to other pods in the system. To this end, a system port SL is provided. A system port includes a corresponding Ethernet port controller EPC and is connected to a system serial link that is connected to a corresponding port in another pod. The routing logic may determine that a packet is intended for routing to another pod in the system and send the packet to the system port SL. A packet may be received on the system port SL from a corresponding system port of another fabric chip in another pod in the system connected via the system serial link and applied to the routing logic.

[0094] It will be appreciated that any type of routing logic may be utilized to route traffic from one external connection of a fabric chip to another connection of the fabric chip (either to another processor chip via an external port or to an attached memory via a memory attached interface). The term "data packet" as used herein refers to a sequence of bits that includes a payload that is transmitted either between processor chips or between a processor chip and a memory attached to a fabric chip. The packet includes information such as a destination identifier and / or memory address for routing purposes. In some embodiments, a destination processor identifier may be included in the packet header. One type of ring routing logic is described in Graphcore's UK Patent Application No. GB2115929.8.

[0095] In some embodiments, there is no identifier for a memory access packet within the POD other than the memory address accessed by the packet. The 32TB address range is divided across a set of fabric chips in the POD, and whether a packet routes to a memory interface or to a port on any given fabric chip depends on the address in the packet and the range of addresses assigned to each fabric chip.

[0096] Packets that are routed between processors (so-called inter-processor packets) differ from memory access packets in that they contain an identifier that identifies which processor (and possibly which tile within the processor) is targeted.

[0097] Both types of packets (memory packets and inter-processor packets) may contain additional information (in the form of an additional identifier (DOMAINID) in the DOMAIN ID field) for routing between pods.

[0098] The fabric chip performs the following routing functions: Incoming packets are first routed to the correct POD by using the DOMAINID field. It is possible that after first entering the target POD, the incoming packet is on the wrong fabric chip and gets routed onwards. · Next, if the incoming packet is a memory access packet, the incoming packet is routed to the correct fabric chip by using the memory address in the packet. Or, if the incoming packet is a processor-to-processor packet, the incoming packet is routed to the correct fabric chip by using the processor identifier (and possibly the tile identifier) ​​in the packet, and once the incoming packet arrives at the correct fabric chip, it is routed to the port that connects to that processor chip.

[0099] Fabric chip 40 may not act as a processor. In other words, fabric chip 40 may act to provide only routing / memory attachment functionality. As a result, fabric chip 40 may not contain suitable processing logic to execute instructions. Thus, packets received by fabric chip 40 are not substantially modified by chip 40 such that the incoming packet is the same as the routing outgoing packet. In some examples, fabric chip 40 may examine packets in order to route them, but does not perform any processing of the packet's contents.

[0100] As described herein, each processing chip can perform a processing or computational function. There are many possible various manifestations of suitable processing chips. Graphcore has developed an intelligence processing unit (IPU) described, for example, in U.S. Patent Application Nos. 15 / 886009, 15 / 886053, and 15 / 886131 [PWF References 408525US, 408526US, and 408527US], the contents of which are incorporated herein by reference. FIG. 10 is a highly schematic diagram of an IPU. The IPU includes multiple tiles 103 on a silicon die, each including a processing unit with local memory. The tiles communicate with each other by using time-deterministic exchanges. Each tile 103 has an instruction storage for holding a local program, an execution unit for executing the local program, a data storage for holding data, an input interface with a set of input wiring, and an output interface with a set of output wiring. The switching fabric 101 (sometimes called a switch or switching fabric) is connected to each of the tiles by respective sets of output wires and is connectable to each of the tiles by sets of input wires via switching circuitry controllable by each tile. A synchronization module (not shown) is operable to generate synchronization signals for switching between the computation phase and the switching phase. The tiles execute their local programs in the computation phase according to a common clock that may be generated on the die or received by the die. At a given time during the switching phase, the tiles may execute a send command from the local program to send a data packet on its output set of connection wires, the data packet being destined for at least one receiving tile but without any destination identifier identifying that receiving tile. At a given switching time, the receiving tile executes a switch control command from the local program to control the switching circuitry to connect the input set of wires to the switching fabric to receive the data packet at the receiving time.The transmission times at which data packets are scheduled to be transmitted from the transmission tiles and the predetermined switching times are governed by a common clock relative to the synchronization signal.

[0101] Time-deterministic exchanges allow efficient transfers between tiles on a die. Each tile has a local memory that provides data storage and command storage. As described herein, the IPU is additionally connected to external memory from which data can be transferred through the fabric chip onto the IPU for use by the tiles.

[0102] The tiles 103 of the IPUs can be programmed such that data packets sent by a SEND command from a local program are either intended to access memory (memory access packets) or are intended to have at their destination another IPU connected within the cluster or system. In these cases, the data packet is sent by the originating tile 103 onto the switching fabric but is not received by the receiving tile within the IPU. Instead, the switching fabric ensures that the tile is provided to the appropriate connector C1, C2, etc. for external communication from the IPU. Packets intended for off-chip communication are generated to include information that defines their final off-chip destination but not the external port from which they are sent. A packet can be sent to an external port by using the principles of time-deterministic switching to identify the external port of the packet when code is compiled for the tile. For example, a memory access packet may identify a memory address. A packet intended for another IPU may include an identifier of the other IPU. This information is used by routing logic on the fabric chip to correctly route off-chip packets generated by the IPU.

[0103] The diagram of Figure 10 shows five example regions of an example IPU chip separated by four boundaries 105 represented by dashed lines. Note that the dashed lines represent abstract boundaries 105 of abstract regions on the processor chip shown for purposes of illustration; that is, the boundaries 105 do not necessarily represent physical boundaries on the IPU chip.

[0104] From top to bottom of the diagram in FIG. 10, the regions separated by boundaries 105 are the upper beachfront, the upper tile region, the switching fabric region, the lower tile region, and the lower beachfront region, respectively.

[0105] It will be understood that the above embodiments have been described by way of example only. Other variations and use cases of the disclosed techniques may become apparent to those skilled in the art given the disclosure herein. The scope of the present disclosure is not limited by the described embodiments, but only by the appended claims.

Claims

1. a single die having a set of external ports; at least one memory attachment interface including a memory controller and configured to attach to an external memory; a fabric core in which routing logic is implemented; 1. A memory attachment and routing chip comprising: The routing logic includes: (i) receiving a first packet of a first type from a first external port of the set of external ports, the first packet being a memory access packet that includes a memory address within a memory address range associated with the memory attachment and routing chip; Detecting the memory address and routing the first packet to the memory attachment interface; (ii) receiving a second packet of a second type, the second packet being an inter-processor packet including a destination identifier that identifies a processing chip external to the memory attach / routing chip, and routing the second packet to a second external port selected from the set of external ports based on the destination identifier; Memory mounting / routing chips.

2. 2. The memory attachment and routing chip of claim 1, wherein the routing logic is configured to examine an incoming packet to determine whether the incoming packet is a memory access packet for routing to the memory controller or an inter-processor packet intended for one of the external ports, and to route the incoming packet based on the determination.

3. Each external port is connected to a link, 3. The memory attachment and routing chip of claim 1 or 2, further comprising at least one link controller formed as circuitry within said die for controlling the transmission of messages on / off said memory attachment and routing chip via said links.

4. the link is a SERDES, 4. The memory attachment and routing chip of claim 3, wherein the link controller comprises circuitry configured to control transmission and reception of packets across links of the SERDES.

5. a first group of the external ports connected to a serial link configured to be attached to a processor chip; 5. The memory attachment and routing chip of claim 1, wherein the routing logic is configured to detect whether the second packet includes a processor identifier for identifying one of a group of external ports.

6. the memory attachment and routing chips are connected in a first computer cluster; 6. The memory attachment and routing chip of claim 1, wherein at least one of the external ports is attached to a cluster connect link configured to connect to another memory attachment and routing chip in a second computer cluster.

7. At least one of the external ports is connected to a system-facing link configured to connect to a switching fabric; 7. The memory attachment and routing chip of claim 1, wherein the routing logic is configured to detect a domain identifier in the first packet or the second packet and route the first packet or the second packet to the system-facing link.

8. 8. The memory attachment and routing chip of claim 1, wherein the set of external ports includes a host port that is connected to a link configured to connect to a host computer.

9. A plurality of memory controllers are included, The memory attachment and routing chip of claim 1 , wherein each memory controller is attached to a dynamic random access memory (DRAM) via a DRAM interface component formed within the die.

10. 10. The memory attachment and routing chip of claim 9, wherein each DRAM interface is constructed according to a JEDEC standard.

11. the die is rectangular; The memory attachment and routing chip of claim 1 , wherein the set of external ports is located along one long side of the die.

12. the memory attachment interface includes a memory attachment connection for connecting the at least one memory controller to an external memory; The memory attachment and routing chip of claim 11 , wherein the memory attachment connections are located along another long side of the die.

13. The memory attachment and routing chip of claim 1 , wherein the die has an aspect ratio greater than 3:

1.

14. the die includes a plurality of bumps for flip-chip attachment of the die to a substrate; 14. The memory attachment and routing chip of claim 1, wherein the bumps include a first group in electrical communication with the external port and a second group in electrical communication with the memory attachment interface.

15. 15. The memory attachment and routing chip of claim 14, wherein a third group of bumps are in electrical communication with a memory controller power supply.

16. 16. The memory attachment and routing chip of claim 14 or 15, wherein the fourth group of bumps is in electrical communication with a chip power supply.

17. 3. The memory attaching and routing chip of claim 1, wherein the routing logic is configured to receive a third packet of a third type including a memory address within a second range of memory addresses outside a range associated with the memory attaching and routing chip, determine a second memory attaching and routing chip associated with the second range of memory addresses, and route the third packet to the second memory attaching and routing chip via one of the external ports.

18. A first memory attachment and routing chip according to any one of claims 1 to 17; A second memory attachment and routing chip according to any one of claims 1 to 17; a first computing device and a second computing device; Including, a first computing device of a second class connected to the first external port of the first memory mounting and routing chip via a first link and connected to the first external port of the second memory mounting and routing chip via a second link; the second computing device of the second class is connected to the second external port of the first memory mounting and routing chip via a third link and to the second external port of the second memory mounting and routing chip via a fourth link; each of the first computing device and the second computing device is configured to execute one or more computer programs and includes a processing circuit coupled to the first link and the second link or the third link and the fourth link for transmitting and receiving messages via the first link and the second link or the third link and the fourth link; computer.

19. 1. A method for routing packets in a computer system including at least one cluster of processor chips, comprising: transmitting a packet including destination information from a designated external port connected to a memory attachment / routing chip of said cluster of a first processor chip; determining, in routing logic of the memory-attached and routing chip, from the destination information, whether the packet is destined for (i) a memory attached to the memory-attached and routing chip, or (ii) a component attached to an external port of the memory-attached and routing chip; based on said determination, in case (i), routing said packet to a memory controller of said memory attach and routing chip, and in case (ii), routing said packet to said external port of said memory attach and routing chip; The method includes:

20. receiving, in the routing logic, a packet including a memory address within a range of memory addresses outside a range associated with the memory attachment and routing chip; determining a second memory attachment / routing chip associated with the second range of addresses; routing the packet to another memory attachment / routing chip via one of the external ports; 20. The method of claim 19, comprising:

Images