DRAM module with data routing logic

Abstract

The memory and routing module (100) includes a substrate (170) and a connection component (160). The connection component (160) is mounted to the substrate (170) and includes a number of pins (161) that connect the module (100) to a corresponding connection component (160) on the motherboard (400). The substrate (170) is connected to a dynamic random access memory (DRAM) chip (110) and a routing chip (140). The routing chip (140) includes a memory controller (142), a number of connections, and routing logic (46). The number of connections includes a first group between the memory controller (142) and the DRAM chip (110) and a second group of connections with the pins (161) of the connection component (160). The routing logic (46) routes data between the second group of connections and the first group of connections.

Description

[Technical Field]

[0001] The present disclosure relates to memory and routing modules and systems including the modules. [Background technology]

[0002] The demand for high performance computing continues to grow, especially in efforts to meet the demands of artificial intelligence / machine learning models that place challenging requirements on computational resources. It is known to address this particular requirement by interconnecting multiple processing chips in clusters, where the processing chips are configured to work collaboratively to meet the demands of processing power needed to process large-scale AI / ML models.

[0003] Another requirement of high performance computing is the ability to access large amounts of memory. To increase memory capacity, attempts have been made to connect external memory to processing nodes in clusters. Such external memory may be connected by a memory bus, which provides an access path between the external memory and the processing nodes. These memory buses may take the form of parallel or serial links. For example, dynamic random access memory (DRAM) may be mounted in dual in-line memory modules (DIMMs) on server racks. These can provide scalable memory capacity on the order of terabytes. Such DIMMs may be mounted vertically on the server rack, and many DIMMs can be stacked together to provide the required memory capacity for a computer. Summary of the Invention [Means for solving the problem]

[0004] The inventors seek to address the coupling problem by providing a cluster of processing chips that have access to large amounts of memory and also allow the processing chips to communicate with each other to increase processing power for specific tasks.

[0005] The present inventors have recognized certain drawbacks associated with the current nature of cluster connectivity. Silicon chips contain circuits arranged two-dimensionally on a die surface that is typically divided into "cores" (e.g., processor cores 2) surrounded by a perimeter or "beachfront" 7 (see FIG. 1). The beachfront is used for input / output (IO) circuitry located at the edge of the chip to reduce signal breakout to packaged pins. For example, beachfront 7 is used to accommodate inter-processor links 6a, 6b and processor-memory links 8a...8d to external memory, as illustrated in FIGS. 2 and 3.

[0006] The area of ​​the beachfront depends on the type and bandwidth of I / O requirements. High-performance computing chips frequently use an approximate maximum manufacturable die size ("full scale") of about 25.5 mm by 32.5 mm, requiring a beachfront depth of about 2 mm on each of the four edges of the die. Using current lithography techniques, a full scale die results in a die core of about 21.5 mm by about 28.5 mm, which is about 74% of the total die area. The computational resources of the die are constrained by this core portion, and the inventors have found that the cost of beachfront area is significant. Figure 1 shows an example die with a full-perimeter beachfront (all four edges) labeled processor cores 2 and 7.

[0007] The present disclosure can help address these problems, and any other problems that will be apparent to the skilled reader from the disclosure contained herein.

[0008] According to a first aspect of the disclosure, a memory and routing module, A substrate; a connection component mounted on the substrate, the connection component including a plurality of pins for mating the module with a corresponding connection component on a motherboard; a dynamic random access memory (DRAM) chip disposed on the substrate; a routing chip mounted on a substrate, a memory controller; a plurality of connections including a first group of connections between the memory controller and the DRAM chips and a second group of connections having pins of a connection component; routing logic configured to route data between the second group of connections and the first group of connections; a routing chip including: The present invention provides a memory and routing module including:

[0009] The DRAM chip may form part of a DRAM chip package that includes a ball grid array (BGA) that connects the DRAM chip package to a substrate.

[0010] The routing chip may include a plurality of bumps that secure the routing chip directly to the substrate by flip-chip securing.

[0011] The memory and routing module may include a plurality of DRAM chips disposed on a substrate. The routing chip may include a plurality of memory controllers. Each memory controller may be connected to a respective DRAM chip of the plurality of DRAM chips via a plurality of connections. The module may include four DRAM chips connected to the routing chip.

[0012] At least one of the plurality of DRAM chips may be disposed on a first side of the substrate, and at least one of the plurality of DRAM chips may be disposed on a second side of the substrate.

[0013] The memory and routing module may include a plurality of routing chips and a plurality of DRAM chips disposed on a substrate, each routing chip including a memory controller connected to each DRAM chip.

[0014] The connection component may be a mezzanine connection component. The connection component may be hermaphroditic. The module may include a plurality of connection components. The module may include a pair of connection components. The connection component may be a ball grid array mounted connection component.

[0015] The routing chip may be mounted on a first side of the substrate. The connection components may be mounted on a second side of the substrate at locations corresponding to the locations of the routing chip. The routing chip may be configured to receive power from power supply components electrically coupled to the connection components, for example, on a motherboard, via the connection components. The module may include a plurality of vias forming electrical paths connecting the connection components to the routing chip to the substrate. The module may not include a power supply component (e.g., a load power feed point). The routing chip may receive power solely via the connection components. The plurality of pins may include a plurality of routing chip power supply pins. The routing chip power supply pins may be configured to receive power from the power supply component. The power supply pins may be disposed on the second side of the substrate at locations corresponding to the locations of the routing chip. In examples including multiple routing chips, the connection component may include a plurality of groups of power supply pins, each group configured to receive power from the power supply component and provide the received power to a respective one of the plurality of routing chips. Each group of power supply pins may be located on the second side of the substrate at a location corresponding to the location of each routing chip.

[0016] The DRAM chips may be low-power double data rate (LPDDR) memory, and the memory controller may include an LPDDR interface.

[0017] The memory controller may be located on the long edge of the routing chip's die, which may face the DRAM chips.

[0018] The second group of connections may include a plurality of processor connections, each configured to attach to a respective processor chip. The routing logic may be configured to route a signal from one of the processor connections to another of the processor connections. The processor connections may be serializer / deserializer (SERDES) connections.

[0019] The plurality of pins may include processor connection pins configured to support a plurality of processor connections. The processor connection pins may include transmit pins and receive pins. The receive pins may be positioned so as not to be adjacent to the transmit pins. In this context, "adjacent" may mean pins adjacent to a given pin, including diagonally. At least some of the receive pins may be separated from the transmit pins by one or more ground pins.

[0020] In examples including multiple routing chips, the multiple pins may include multiple groups of processor connection pins, each group configured to support multiple processor connections of one of the multiple routing chips. Each group of processor connection pins may include a transmit pin and a receive pin. The receive pins in a group may be arranged so as not to be adjacent to the transmit pins in the group. At least some of the receive pins in a group may be separated from the transmit pins in the group by one or more ground pins. A transmit pin of one of the multiple groups of processor connection pins may be arranged so as not to be adjacent to a receive pin of another of the multiple groups of processor connection pins. Suitably, the transmit pins of each group are arranged so as not to be adjacent to a receive pin of any other of the multiple groups.

[0021] The plurality of pins may include a DRAM chip power supply pin configured to receive power for the DRAM chip. The DRAM chip power supply pin may be disposed along an edge of the connection component. The edge may be an edge of the connection component closest to the DRAM chip.

[0022] The substrate may be a package substrate including a plurality of conductive lines, and the first group of connections and the second group of connections may be via the plurality of conductive lines.

[0023] According to a second aspect of the disclosure, there is provided a system including a memory and routing module as described herein and a plurality of processor chips connected to the memory and routing module via connection components.

[0024] The processor chips may not be in direct electrical communication with each other, but may be configured to communicate with other processor chips via routing chips.

[0025] Multiple processor chips may perform time-deterministic processing.

[0026] The system may include a motherboard to which the modules can be attached. The motherboard may be configured for attachment of multiple processor chips. The system may include multiple modules. The motherboard may include power supply components that provide power to the modules.

[0027] The disclosure also extends to methods of routing data using the memories and routing modules described herein.

[0028] Further optional features of the system and corresponding method of the second aspect are described above with respect to the modules of the first aspect and may be combined in any combination.

[0029] According to a fourth aspect of the disclosure, a package substrate for receiving a flip-chip mounted semiconductor chip in a module; a first flip-chip mounted semiconductor chip mounted on a package substrate; a first ball grid array mounted packaged semiconductor chip mounted on a package substrate, the first flip chip mounted semiconductor chip and the first ball grid array mounted semiconductor chip being in electrical communication with each other; a connection component mounted on the package substrate, the connection component including an electrical coupling for coupling the package substrate to a corresponding connection component on a motherboard; Including, the package substrate includes a first ball grid array mounted semiconductor chip mounted to the package substrate and a plurality of conductive lines coupling the first flip chip mounted semiconductor chip to the connecting components; Provide a module.

[0030] The first ball grid array mounted semiconductor chip may be a dynamic random access memory (DRAM) chip. The DRAM chip may be a low-power double-data-rate (LPDDR) chip. The module may include a plurality of ball grid array mounted semiconductor chips.

[0031] The package substrate may be a monolithic package substrate, and at least some of the plurality of ball grid array mounted packaged semiconductor chips may be disposed on the monolithic package substrate.

[0032] The module may include a plurality of flip-chip mounted semiconductor chips mounted on a package substrate. The plurality of flip-chip mounted semiconductor chips may be in electrical communication with a plurality of ball grid array mounted semiconductor chips. Each flip-chip mounted semiconductor chip may be in electrical communication with a subset of the plurality of ball grid array mounted semiconductor chips. Each flip-chip mounted semiconductor chip may be in electrical communication with four ball grid array mounted semiconductor chips. The module may include four flip-chip mounted semiconductor chips.

[0033] The first flip-chip mounted semiconductor chip may include routing logic configured to route data between connection components mounted to the package substrate and the first ball grid array mounted packaged semiconductor chip.

[0034] The conductive traces coupling the first flip-chip mounted semiconductor chip to the connection component may include a plurality of processor connections. The first flip-chip mounted semiconductor chip may include routing logic configured to route data from one of the processor connections to another of the processor connections. The processor connections may include serial connections, such as serializer / deserializer (SERDES) links.

[0035] A first flip-chip mounted semiconductor chip may be mounted to a first side of the package substrate. A first ball grid array mounted semiconductor chip may be mounted to the first side of the package substrate. A first ball grid array mounted semiconductor chip may be mounted to a second side of the package substrate.

[0036] The module may include a second ball grid array mounted semiconductor chip mounted on a second side of the package substrate. The package substrate may include a plurality of vias forming electrical paths electrically connecting the second ball grid array mounted semiconductor chip to the first flip-chip mounted semiconductor chip. At least one of the vias may be located under the first flip-chip mounted semiconductor chip.

[0037] A first flip-chip mounted semiconductor chip may be mounted on a first side of the package substrate, and a connection component may be mounted on a second side of the substrate at a location corresponding to the location of the first flip-chip mounted semiconductor chip. The first flip-chip mounted semiconductor chip may be configured to receive power from a power supply component electrically coupled to the connection component via the connection component. The module may include a plurality of vias forming an electrical path connecting the connection component to the package substrate and the first flip-chip mounted semiconductor chip. The module may not include a power supply component (e.g., a load power feed point). The first flip-chip mounted semiconductor chip may receive power solely via the connection component.

[0038] The first side and the second side of the substrate may be opposite sides of the substrate.

[0039] The package substrate may include multiple layers suitably formed on a core. At least two of the layers may include conductive traces that transmit signals between the first flip-chip mounted semiconductor chip and the first ball grid array mounted packaged semiconductor chip, and between the first flip-chip mounted semiconductor chip and connecting components. The substrate may be a high density interconnect (HDI) substrate.

[0040] The connection component may be a mezzanine connector. The connection component may include a plurality of pins. The connection component may be hermaphroditic. The module may include a plurality of connection components. The module may include a pair of connection components. The connection component may be a ball grid array mounted connection component.

[0041] According to a fifth aspect of the disclosure, there is provided a system including a module as described herein and a plurality of processor chips connected to the module via a connection component.

[0042] According to a sixth aspect of the disclosure, there is provided a system including a module as described herein and a motherboard to which the module is mountable. The motherboard may be configured for mounting a processor chip. The system may include the processor chip. The system may include multiple processor chips and multiple modules. The motherboard may include power supply components that provide power to the module.

[0043] According to a seventh aspect of the disclosure, 1. A method of manufacturing a module, comprising: providing a package substrate; forming a plurality of conductive traces on a package substrate; mounting a first semiconductor chip to a package substrate by flip-chip mounting; attaching a ball grid array packaged semiconductor chip to a package substrate; attaching connection components to the package substrate, the connection components including electrical connections that couple the package substrate to corresponding connection components on a motherboard; Including, a plurality of conductive lines electrically connecting the first semiconductor chip to the ball grid array packaged semiconductor chip and the connecting components; A method is provided.

[0044] The method may include heating the module and attaching a first semiconductor chip before attaching a ball grid array packaged semiconductor chip or connecting component. The method may include underfilling the first semiconductor chip.

[0045] The method may include the steps of heating the module and attaching the ball grid array packaged semiconductor chips before attaching the connection components, and heating the module and attaching the connection components.

[0046] The method may include forming a plurality of vias in a package substrate, mounting a first semiconductor chip to a first side of the package substrate, and mounting at least one of the ball grid array packaged semiconductor chip or the connecting component to a second side of the package substrate. At least one of the conductive lines may pass through the vias and connect the first chip to the ball grid array packaged semiconductor chip or the connecting component.

[0047] The method may include forming a plurality of layers on a package substrate core, at least two of the layers including conductive lines that transmit signals between the first semiconductor chip and the ball grid array packaged semiconductor chip and connecting components.

[0048] Further optional features of the method of the seventh aspect are described above in relation to the modules of the fourth aspect and may be combined in any combination.

[0049] Furthermore, the features of the modules, systems, and methods described in the first to third aspects may be combined with the modules, systems, and methods of the fourth to seventh aspects.

[0050] In certain embodiments of the present invention, any processor chip may access any memory attached to any of the fabric chips in the computer cluster. Memory access may be via a high-speed serial link. Furthermore, any processor may exchange packets with any other processor in the computer via routing logic in the fabric chips.

[0051] In certain aspects of the present invention, the inventors enable clusters of processing chips in multiple tiers.

[0052] In certain aspects of the present invention, each processing chip itself improves processor core area for a particular size of substrate.

[0053] Another requirement imposed by high performance computing is the ability to have high bandwidth to large capacity memories. Currently, so-called high bandwidth memories (HBMs) are implemented by providing memory within the physical structure of the processing node itself. That is, the memory is provided in close proximity to the processing chips that are mounted on a silicon substrate within the package that forms the processing node. In effect, the HBM is butted up against the processing chips on the silicon substrate, so as to be as physically close as possible to the processing chips that provide the processing functionality. In this way, high bandwidth is achieved, but there are limitations to memory capacity based on the physical size of the memory that can be accommodated in this type of structure. Furthermore, such HBMs are expensive to manufacture.

[0054] In the fields of artificial intelligence (AI) and machine learning (ML), mathematical models can be extremely large, requiring extremely large amounts of memory to accommodate them. As the size of the models increases, the cost of implementing HBM also increases.

[0055] Currently, the lack of availability of large-capacity, high-bandwidth memory places constraints on the size and nature of models available to machine learning / artificial intelligence computers. In particular, the knowledge capacity of a model is a function of the amount of memory that is reasonably accessible. In some embodiments of the present invention, a portion of the beachfront may no longer be used for connecting to external memory and may be made available to the HBM.

[0056] For a better understanding of the present invention and to show how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0057] [Figure 1] FIG. 1 is a simplified block diagram of a chip connected to a memory. [Figure 2] 1 is a schematic diagram of multiple interconnected chips. [Figure 3] FIG. 1 is a block diagram of multiple processor chips connected using a switch core. [Figure 4]1 is a simplified block diagram of a processor chip with a reduced beachfront. [Figure 5] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips. [Figure 6] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips having a higher ratio of fabric chips to processor chips. [Figure 7] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips, each processor chip with high-bandwidth memory. [Figure 8] 1 is a simplified block diagram of a computer including a set of interconnected clusters. [Figure 9] FIG. 1 is a simplified block diagram of a fabric chip. [Figure 10] 1 is a schematic diagram of an example processor chip. [Figure 11a] FIG. 1 is a top perspective view of an example memory and routing module. [Figure 11b] 11b is a bottom perspective view of the example memory and routing module of FIG. 11a. [Figure 12] 12 is a schematic diagram of the top view of the example memory and routing module of FIG. 11. [Figure 13] 13 is a schematic diagram of the underside of the example memory and routing module of FIGS. 11 and 12. FIG. [Figure 14a-14b] 14 is a schematic diagram of the substrate of the example memory and routing module of FIGS. 11-13. FIG. [Figure 15] 1 is a schematic diagram of an example layout of a fabric chip. [Figure 16] 14 is a schematic diagram illustrating pins of connection components of the example memory and routing modules of FIGS. 11-13. FIG. [Figure 17] 1 is a simplified flowchart of an example method for manufacturing a memory and routing module. [Figure 18] 14 is a simplified cross-sectional view of the example memory and routing module and motherboard of FIGS. 11-13. FIG. [Figure 19]14 is a schematic diagram illustrating pins of another example of a pair of connection components usable in the memory and routing modules of FIGS. 11-13. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0058] In the drawings, corresponding reference characters indicate corresponding components. Those skilled in the art will appreciate that the elements in the drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the drawings may be exaggerated relative to other elements to help facilitate a better understanding of the various example embodiments. Furthermore, to facilitate viewing of these various example embodiments, common, well-understood elements that are useful or necessary in commercially viable embodiments are often not shown.

[0059] There are various known methods for forming clusters of processing chips by interconnecting the processing chips with each other.

[0060] Figure 1 shows an example of a processor chip intended to be connected in a processing cluster. Processing chip 1 includes a processor core 2 (shown cross-hatched) implemented on a silicon die 4. It is convenient to distinguish between a beachfront area, where external links are provided, and a core area for the processing circuitry of the processor core. The beachfront area includes inter-processor links 6a, 6b, and processor-memory links 8a, 8b, 8c, 8d (shown in Figure 1) connected to respective DRAMs 10a, 10b, 10c, 10d.

[0061] Figure 2 illustrates a cluster of four processing units of the type illustrated in Figure 1, with fully direct connectivity of inter-processor links. Each processor core 2a, 2b, 2c, and 2d is shown with three external connectors 5 along its top edge and three external connectors 5' along its bottom edge. In the cluster of Figure 2, each processor core is connected to each other processor core by two external connection links attached to the external connectors in an exemplary manner. See, for example, links L and L' connecting processor core 2a to processor core 2d. Note that there remains a need for a dedicated processor chip-to-memory bus to accommodate DRAM.

[0062] This is just one example of inter-processor connectivity in a cluster.

[0063] An alternative method of connecting processor chips together in a cluster is to use a switch fabric. Figure 3 is a schematic diagram illustrating the connection of four processor cores 2a, 2b, 2c, and 2d to each of two switch cores 12a and 12b. Each switch core can be configured to route traffic between specific processor cores under program control. In this arrangement, each processor has access to its own externally connected DRAM.

[0064] In the above examples, each processing chip has access to memory. In some previous examples, that memory may be externally attached memory connected to each processor core in the cluster and / or high-bandwidth memory (HBM) connected within the processor package. In either case, the memory attachment uses the "beachfront" of the die.

[0065] In a specific embodiment of the present disclosure, a computer includes multiple processor chips and fabric chips arranged in a cluster. Within the cluster, each processor chip is connected to all fabric chips, and each fabric chip is connected to all processor chips in an exhaustive bifurcated configuration. There are no direct connections between the fabric chips themselves within the cluster. Furthermore, there are no direct connections between the processor chips themselves. Each fabric chip has routing logic configured to route incoming packets from one processor chip connected to the fabric chip to another processor chip. Furthermore, each fabric chip has means for attaching to external memory. The routing logic can route packets between a processor connected to the fabric chip and the memory attached to the fabric chip. The fabric chip itself includes a memory controller that performs memory control functions that manage memory accesses to and from the memory attached to the fabric chip.

[0066] In certain embodiments further described herein, clusters of processing chips and fabric chips may themselves be interconnected to form larger computer systems. Each processor chip in a cluster may access any of the memory attached to any of the fabric chips in the cluster. This greatly expands the memory capacity available to any particular processor chip.

[0067] The connection configurations described herein have the additional advantage that, in certain embodiments, it is not necessary to use the entire edge of the processor die to surface the external connections.

[0068] The inventors have found that limiting the beachfront required for connections to less than all four edges of the die advantageously frees up more silicon for fabrication of the processing "core." For example, if only the short edges of a full reticle die are used for I / O, the area available for processor cores on the chip increases to about 88% of the total die area, about 19% more than for four sides. Figure 4 shows an example of such a chip 1' in which the vertical edges cannot accommodate the beachfront, and the top and bottom edges have beachfronts 7a and 7b, respectively.

[0069] Prior art processing cluster connectivity requirements include a beachfront around the entire perimeter (e.g., as shown in FIG. 1). Particular implementations of the present connectivity configuration described herein enable the use of processor dies that have beachfronts only on the top and bottom edges, but not on the vertical edges (as shown in FIG. 4).

[0070] In accordance with the presently described embodiment of the present invention, multiple processors are connected in a cluster using one or more "fabric chips." Each fabric chip provides access to external memory (e.g., DRAM) and also provides routing for inter-processor traffic. Referring now to FIG. 5, FIG. 5 illustrates four processor chips 20a, 20b, 20c, and 20d. Each processor chip includes a processor core area 22a, 22b, 22c, and 22d extending to each vertical edge of the chip. Each processor chip has an upper beachfront area 30a and a lower beachfront area 30b (only chip 20a is shown). Upper beachfront area 30a has a set of external port connections C1, C2, C3, and C4 (labeled only for processor chip 20a). Each processor chip can also be seen to have four external port connections in its upper beachfront area. Similarly, the lower beachfront area of ​​each processor chip has four external port connections labeled C5, C6, C7, and C8. Note that the bottom set of external port connections is labeled only on processor chip 20a. It will be appreciated that the other processor chips similarly each have a set of external port connections in their bottom beachfront regions.

[0071] The cluster of FIG. 5 further includes eight "fabric chips." Each fabric chip includes fabric cores 40a, 40b,..., 40h. Each fabric chip has a lower beachfront region 44a...44h with a set of external ports. These external ports are provided on fabric chip 40a only at port connections labeled FC1, FC2, FC3, and FC4. It can be seen that each fabric chip has a corresponding set of external ports on its lower beachfront region. The upper beachfront region of each fabric chip includes one or more memory attachment interfaces that allow the fabric chip to connect to one or more memories, illustrated in FIG. 5 as DRAMs 10a, 10b, 10c, 10d...,..., 10p. For example, fabric core 40a shown in FIG. 5 is connected to two DRAMs 10a and 10b by appropriate memory attachment interfaces provided on the upper beachfront 46a of the fabric chip. Other large capacity memories, such as double data rate DRAM (DDR), and more recent manifestations of that DRAM, such as low power DDR (LPDDR), may also be connected. The high-bandwidth connections between the processor chips and fabric chips in the cluster are "exhaustively bifurcated," meaning that each processor chip is connected to every fabric chip and each fabric chip is connected to every processor chip. The connections are via links (e.g., L1) between processor ports at port connections (e.g., C1) and fabric chip ports at port connections (e.g., FC1). However, it should be noted that in the illustrated example, there are no direct high-bandwidth connections between processor chips or fabric chips within the cluster. Furthermore, in the illustrated example, there is no externally attached memory directly connected to each processor (although there may be high-bandwidth memory within the chip package (see below)). Each fabric chip provides routing functionality that provides paths between all pairs of processors and between each processor and the memory attached to the fabric chip.

[0072] Furthermore, the links can be manifested in any suitable manner. Each link can be connected or reconnected to a different port to set up the computer configuration. Once the computer configuration is set up and operational, the links are not multiplexable and do not fan-in or fan-out. That is, instead, there are no intermediate switches directly connecting a port on the processor to an end port on the fabric chip. Any packet transmitted on the link is received by a port at the other end of the fixed link. The link is advantageously bidirectional; although bidirectionality is not a mandatory requirement, the link is preferably capable of operating in both directions simultaneously. One particular category of communication link is a SERDES link, which has power requirements that are independent of the amount of data transmitted on the link or the time consumed transmitting that data. SERDES is an acronym for serializer / deserializer, and such links are known. For example, a SERDES link may be implemented using twisted-pair wire. Transmitting a signal on the wire of such a link requires power applied to the wire, and the voltage is varied to generate the signal. SERDES links have the characteristic that there is a fixed power consumption relative to the bandwidth capacity on the SERDES link, regardless of whether it is used or not. This is due to the need to provide clocking information about the link by constantly switching the current or voltage state of the line, even when no data is being transmitted. As is known, data is transmitted by holding the state of the line and indicating a logic "0" or a logic "1." SERDES links are implemented at each end by circuitry that connects link layer devices to the physical link (e.g., copper wire). This circuitry is sometimes called the PHY (Physical Layer). In this example, Layers 1 and 2 of the Ethernet protocol are used to transmit packets over the link. However, it will be appreciated that any data transmission protocol can be used.

[0073] There are several advantages to the computer described herein.

[0074] There is no longer a need to dedicate a fixed percentage of the processor beachfront (and therefore I / O bandwidth) to a fixed amount of memory or inter-processor connectivity. All processor I / O bandwidth flows through the fabric chip and is available on demand for either purpose (memory or inter-processor).

[0075] Under some popular models of microprocessor computing, such as massively synchronous parallelism (BSP), peak DRAM bandwidth and peak inter-processor bandwidth usage may not be simultaneous. Therefore, total bandwidth requirements can be met with a smaller processor beachfront, affording a larger core area to the processor chip. BSP itself is known in the art. According to BSP, each processing node performs a computation phase and an exchange phase (also called a communication or message-passing phase) in alternating cycles. The computation phase and exchange phase are performed by the processing chip that executes instructions. During a computation phase, each processing unit locally performs one or more computational tasks but does not communicate any results of these computations to other processing chips in the cluster. During an exchange phase, each processing chip can exchange one or more processing results from a previous computation phase with one or more other processing chips in the cluster. It should be noted that different processing chips can be assigned to different groups for synchronization purposes. According to the principles of BSP, barrier synchronization is performed at the junctions transitioning from the computation phase to the exchange phase, or from the exchange phase to the computation phase, or at both junctions. That is, require all processing chips in a group to complete each computation step before any of the group can proceed to the next switching step, or require all processing chips in a group to complete each switching step before any processing chip in the group can proceed to the next computation step, or perform both of these conditions. This sequence of switching and computation steps is repeated over many cycles. In BSP terminology, each iterative cycle of switching and computation steps is sometimes called a "superstep."

[0076] This has the practical effect that there are situations where there is no simultaneous use of all links required to access memory (for the purposes of the computation phase) and the links used to exchange data between processing chips in the exchange phase. As a result, there is maximum efficient use of fixed links without compromising memory access times and inter-processor exchange delays. Nevertheless, it will be appreciated that the embodiments described herein have applications other than when used with BSP or other similar synchronization protocols.

[0077] Links can be dynamically deactivated while not in use to effectively conserve power. However, the runtime and non-deterministic nature of machine learning applications generally make dynamic activation during program execution problematic. As a result, the inventors have determined that it is better to take advantage of the fact that link power consumption is essentially constant for any particular configuration, and therefore the best optimization is to maximize physical link usage by maintaining as much parallel inter-processor and processor-memory activity as possible.

[0078] All memory in the cluster is accessible to each processor without having to go around in circles through another processor. This shared memory arrangement can benefit software efficiency.

[0079] In the example shown in FIG. 5, there are two "ranks" of fabric cores, one mounted on each top and bottom edge of the processor chip. The top rank includes fabric cores 40a...40d, each connected to a respective processor core by a respective link. For example, processor core 20a is connected to fabric core 40a by link L1, to fabric core 40b by link L2, to fabric core 40c by link L3, and to fabric core 40d by link L4. The bottom rank includes fabric cores 40e...40h. Fabric core 40a is also connected to each processor core 20a...20d by corresponding links (shown in the drawing but not labeled for clarity). There is no use of vertical processing chip edges for beachfronting.

[0080] However, there are different design choices within the overarching concept: for example, the vertical edge of the processor can be used to provide more bandwidth to the fabric chip, and all links coming out of the beachfront of the processor chip can be passed to one rank or three ranks of the fabric chip, etc.

[0081] The number of fabric chips in each rank may differ from the number of processor chips. What remains important to achieving the benefits of the present invention is maintaining exhaustive bipartite connectivity between the processing chips and the fabric chips, with routing functionality and external memory access provided by the fabric chips.

[0082] FIG. 6 shows a specific example of connecting four processing chips to eight fabric chips in an upper rank and eight fabric chips in a lower rank. Each processing chip is connected to 16 fabric chips. Take processor chip 20a as an example. This processor chip has eight upper rank connectors C1, C1', C2, C2', C3, C3', C4, and C4', each connected to a link connector on each of the two fabric cores, labeled 40a and 40a' in FIG. 6. The processor chip also has eight lower rank connectors C5, C5', C6, C6', C7, C7', C8, and C8', each connected to a link connector on each of the eight fabric chips in the lower rank. Each fabric chip is connected to four processor chips.

[0083] It should be noted that the use of external connectors to provide exhaustive bifurcated connectivity in a cluster according to examples of the present invention does not preclude the presence of other I / O ports on the processor chip or fabric chip. For example, a particular one of the processor chips or fabric chips in a cluster may be provided with an I / O port that allows connectivity between multiple clusters, to a host device, etc. In one embodiment described with reference to Figures 8 and 9, the fabric chip provides this additional connectivity.

[0084] It should also be noted that additional memory can be mounted directly on the processor chip, for example, along a vertical edge. That is, additional high-bandwidth memory (HBM) may be provided in close proximity to the processing chips mounted on the silicon substrate within the package forming the processing node. In practice, the HBM is butted against the processing chips on the silicon substrate, so as to be as physically close as possible to the processing chips providing the processing functionality. For example, the high-bandwidth memory (HBM) can be mounted on the processor chip, while the high-capacity memory can be mounted on the fabric chip, combining the advantages of both memory types in a cluster. FIG. 7 illustrates an embodiment in which high-bandwidth memory (HBM) modules 26 are mounted on the east and west edges of each processor chip 20'a, 20'b, 20'c, and 20'd. In other respects, the computer illustrated in FIG. 7 has the same connections as the computer described in FIG. 5. The HBM 26 may be mounted by a short parallel connection of memory buses formed on the substrate or by using a silicon bridge in the package substrate.

[0085] In the exemplary computer described herein, the processor chip 20 is not intended to be deployed standalone. Instead, the processor chip is deployed within a computer cluster, where the processor chip is supported by one or more fabric chips 40. The processor chips 20 connect to each other via the fabric chips 40, enabling the use of all processor chip links L1, L2, etc. for simultaneous use as inter-processor links and memory access links. In this manner, the computer provides a larger capacity, faster memory system than existing computer systems. In current computer systems, providing large capacity, high-bandwidth memory is becoming increasingly expensive. Furthermore, limitations remain on the processing power that can be obtained while providing high-bandwidth memory access and large capacity memory. The present computer can overcome these limitations.

[0086] By providing routing logic on the fabric chip, the processor chip does not need to have routing logic for external routing functions, which frees up silicon area to maximize input / output bandwidth per processor chip and also maximizes the area available for processing circuitry within the processor core.

[0087] By placing link ports along the north and south edges, this frees up the east / west edges so that processor cores can extend to the east / west edges, thus maximizing processing power, or leaving the east / west edges free for high bandwidth memory integration.

[0088] A computer may operate in different topologies. In one example, a group of four processor chips and eight fabric chips (e.g., as illustrated in FIG. 5) may comprise a cluster. Within a cluster, each group of four fabric chips connected to one of the processor chip edges is referred to herein as a rank. A cluster of eight includes two ranks.

[0089] A pod may contain multiple clusters. Clusters may be interconnected within a pod using processor-to-processor links in the fabric chip. Pods may be interconnected to each other using pod-to-pod links in the fabric chip. This is shown in more detail in FIG. 9, which illustrates an example fabric chip.

[0090] FIG. 8 is a schematic diagram of a system topology and hierarchy according to one embodiment. FIG. 8 illustrates a number of pods P1, P2, P3, ..., Pn (labeled POD16). In the example of FIG. 8, n = 8, but it is readily apparent that a different number of pods can be connected to a computer system using the techniques described herein. One of the pods, pod P1, is shown in detail. Pod P1 includes four clusters Q1, Q2, Q3, and Q4. In the example of FIG. 8, each cluster includes four processor chips 20a, 20b, 20c, and 20d that share 32 fabric chips. Fabric chips 40 are labeled in FIG. 8, with Q4 indicating, for example, that fabric chip 40 (Q4) is in cluster Q4. As shown in FIG. 8, in each cluster Q1, Q2, Q3, and Q4, the four processor chips 20a, 20b, 20c, and 20d are connected to the 32 fabric chips in an exhaustive bisection arrangement. That is, as described above, each fabric chip in a cluster is connected to all four processor chips, and each processor chip is connected to all 32 fabric chips. Each processor chip has 32 port connections C1, C2, ..., C32 (16 on the top edge and 16 on the bottom edge). As illustrated in Figure 9, in a specific embodiment, each port connection provides three bidirectional serial links, forming a total of 96 processor links. Each set of 12 processor external links (out of the 96 links) connects to each set of four fabric chips (three processor links to each fabric chip port connection FC). Thus, eight sets of 12 links connect to eight sets of four fabric chips within a cluster. Four clusters Q1, Q2, Q3, and Q4 are grouped to form pods, with 32 fabric chip pod-to-pod links within each cluster. Each cluster sends three bundles to each of the 32 links, and each bundle connects to each of the other three clusters. A bundle of pod-facing links between two clusters includes one link between each of the 32 corresponding peer fabric chips in the two clusters. A particular pod-facing link may be connected to a third-party Ethernet switch.

[0091] FIG. 9 is a simplified block diagram of the components on the fabric chip 40. As shown in FIG. 9, routing logic 46 is connected to the DDR interface block 48 for transferring data packets between the DDR interface block 48 and other ports. Furthermore, routing logic 46 is attached to each processor-facing link port. Each port includes an Ethernet port controller EPC. Routing logic is attached to the Ethernet port controller for the pod-facing port and the Ethernet port controller for the system-facing link. Furthermore, routing logic 46 is attached to the PCI complex for interfacing to the host system. PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high-speed computers.

[0092] FIG. 9 illustrates an example of a fabric chip that can be used to configure a computer by connecting computer clusters together in a hierarchical manner, as well as inter-processor and processor-to-memory communication. The components of the fabric chip used to perform inter-processor and processor-to-memory communication are described first. Each fabric core port connection includes three serial links. Each serial link includes a port with an Ethernet port controller (EPC). As noted, these links may be SERDES links, e.g., twisted pair wires, that enable serial packet communication.

[0093] For clarity, not all components in FIG. 9 are illustrated with associated reference. Each fabric core connection FC1, FC2, FC3, and FC4 has a configuration as described herein with reference to a fabric core port connection FC2 that connects to a second processor (e.g., processor 20b in FIG. 6). Fabric connection FC2 includes three links L2a, L2b, and L2c, each including an Ethernet port controller EPC2a, EPC2b, or EPC2c, respectively. It should be noted that in other embodiments, a single physical link may be provided, or a different number of physical links may be provided for each fabric chip connection FC. Thus, it should be noted that the link labeled L2 in the previous figures may include three individual serial links (e.g., L2a, L2b, and L2c). The routing logic 46 in the fabric chip 40 may be implemented as a ring router, a crossbar router, or in any other manner. The fabric chips are further connected to external memory (e.g., DRAMs 10A, 10B, etc.) (not shown in FIG. 9). While two DRAMs are shown in previous figures, in the embodiment of FIG. 9, the fabric chip connects to four DRAMs. To make this connection, the fabric chip includes four DRAM interface blocks DIB1, DIB2, DIB3, and DIB4, each associated with four DDR subconnect layers DDR sub1, DDR sub2, DDR sub3, and DDR sub4. Each DDR interface block DIB 48 incorporates a memory controller that manages access to the memory attached to the block. While one memory attachment interface 44 is shown in FIG. 9, it is understood that each DDR sublayer has a respective memory attachment interface that attaches to external DRAM. Routing logic 46 is configured to route memory access packets received from the attached processor core to an addressed one of the data interface blocks DIB1-DIB4. Furthermore, routing logic 46 is configured to route from one attached processor chip to another attached processor chip via each fabric chip port.In certain embodiments, the routing logic prevents memory packets (e.g., memory access response packets) from being routed from one memory attached interface to another. In such embodiments, the memory response packets may simply be routed to the processor chip via the correct port attached to the routing logic 46. For example, an incoming packet on link L2a of fabric core port connection FC2 is routed to an addressed port connected to the routing logic 46 based on the packet's routing information. For example, if the packet is intended to be routed to processor 20c, the routing logic 46 identifies processor 20c from the packet's routing information and causes the packet to exit via the Ethernet port controller onto the link attached to processor 20c.

[0094] If the packet is a memory access packet, the routing logic routes the packet to the appropriate DDR interface block based on the packet's memory address. Note that in this embodiment, each DDR interface block DIB1...DIB4 includes four memory access channels. It is understood that any number of memory access channels can be provided by each interface block DIB1...DIB4. The memory access channels are managed by a memory controller in each data interface block DIB1...DIB4.

[0095] As mentioned above, in the example shown in FIG. 9 , fabric chip 40 has additional components that enable the computer to be configured into interconnected clusters. To this end, the fabric chip includes a pod-facing port connection PL. The pod-facing port connection PL includes three ports, each with an Ethernet port controller Pa, Pb, or Pc connected to a respective link. The routing logic detects a packet whose packet information indicates that the packet should not be routed to a processor in this cluster but should instead be routed to a processor in another cluster, and routes the packet to one of the pod-facing ports. It should be noted that the pod-facing port connection PL can send packets to or receive packets from corresponding pod-facing ports on fabric chips in another cluster.

[0096] The fabric chip of Figure 9 can also route packets to another pod in the system. To this end, a system port SL is provided. The system port includes a corresponding Ethernet port controller EPC and is connected to a system serial link connected to a corresponding port in another pod. The routing logic may determine that the packet is intended for routing to another pod in the system and send the packet to the system port SL. The packet may be received on the system port SL from a corresponding system port of another fabric chip in another pod in the system connected via the system serial link and applied to the routing logic.

[0097] It will be appreciated that any type of routing logic can be utilized to route traffic from one external connection of a fabric chip to another connection of the fabric chip to another processor chip via an external port or to attached memory via a memory attached interface. The term data packet as used herein means a bit string containing a payload to be transmitted between processor chips or between a processor chip and memory attached to a fabric chip. The packet contains information (e.g., a destination identifier and / or memory address for routing). In some embodiments, the destination processor identifier may be included in the packet header. One type of ring routing logic is described in Graphcore's UK Patent Application No. 2115929.8.

[0098] As described herein, each processing chip can perform a processing or computational function. There are many possible different manifestations of a suitable processing chip. Graphcore has developed an intelligent processing unit (IPU), as described, for example, in U.S. Patent Application Nos. 15 / 886,009, 15 / 886,053, and 15 / 886,131 [PWF Refs. 408525US, 408526US, and 408527US], the contents of which are incorporated herein by reference. Figure 10 is a schematic diagram of an IPU. The IPU includes multiple tiles 103 on a silicon die, each including a processing unit with local memory. The tiles communicate with each other using time-deterministic exchanges. Each tile 103 has an instruction store for holding a local program, an execution unit for executing the local program, a data store for holding data, an input interface with a set of input lines, and an output interface with a set of output lines. The switching fabric 101 (sometimes called a switch or exchange fabric) is connected to each tile by a respective set of output lines and to each tile by a respective set of input lines via switching circuits connectable by each tile. A synchronization module (not shown) is operable to generate synchronization signals and switch between the computation phase and the exchange phase. The tiles execute their local programs in the computation phase according to a common clock that can be generated on the die or received by the die. At predetermined times in the exchange phase, the tiles execute send instructions from their local programs and send data packets to an output set of connection lines, the data packets being destined for at least one recipient tile but not having a destination identifier identifying that recipient tile. At predetermined switch times, the recipient tile executes switch control instructions from its local program to control the switching circuits to connect an input set of lines to the switching fabric and receive the data packets at receive times. The transmit times at which data packets are scheduled to be sent from the sending tile and the predetermined switch times are controlled by the common clock relative to the synchronization signal.

[0099] The time-deterministic exchange allows for efficient transfers between tiles on the die. Each tile has its own local memory that provides data storage and instruction storage. The IPU is further connected to external memory that can transfer data to the IPU for use by the tile via the fabric chip, as described herein.

[0100] The tiles 103 of an IPU may be programmed so that data packets sent by a SEND instruction from a local program are intended to access memory (memory access packets) or have as their destination another IPU connected in the cluster or system. In such cases, the data packets are sent to the switching fabric by the originating tile 103 but are not picked up by the receiving tile within the IPU. Instead, the switching fabric equips the tile with the appropriate connectors C1, C2, etc. for external communication from the IPU. Packets for off-chip communication are generated to include information defining the final off-chip destination rather than the external port to which they should be sent. Packets may be sent to an external port using the principle of time-deterministic switching to identify the external port for the packet when code is compiled against the tile. For example, a memory access packet may identify a memory address. A packet intended for another IPU may include an identifier for the other IPU. This information is used by routing logic on the fabric chip to correctly route off-chip packets generated by the IPU.

[0101] 10 shows five example regions of an example IPU chip separated by four dashed boundary lines 105. Note that the dashed lines represent abstract boundaries 105 of abstract regions on the processor chip shown for illustrative purposes, and that the boundaries 105 do not necessarily represent physical boundaries on the IPU chip.

[0102] From top to bottom of the diagram in FIG. 10, the regions separated by boundary lines 105 are the upper beachfront, the upper tile region, the switching fabric region, the lower tile region, and the lower beachfront, respectively.

[0103] The above presents a logical arrangement of the computer system described herein, including processor cores or chips 20, fabric chips 40, and DRAM 10. Below, the physical layout and configuration of some elements of the computer system are described in more detail.

[0104] 11a-14b, there is shown a memory and routing module 100 according to an example of the disclosure.

[0105] Module 100 includes multiple fabric chips 140, multiple DRAMs 110, and two connection components 160. Fabric chips 140 and DRAMs 110 correspond to fabric chips 40 and DRAMs 10 described above. That is, fabric chips 140 and DRAMs 110 described below may incorporate the features described above in terms of fabric chips 40 and DRAMs 10. Fabric chips 140 on module 100 include a memory controller, described in more detail below, that accesses DRAMs 110.

[0106] The fabric chips 140, DRAM 110 and connection components 160 are mounted on a substrate 170 in the form of a flat plate. The plate has dimensions of approximately 80 mm x 70 mm, e.g., approximately 5300-5400 mm. 2 The substrate may be 77 mm by 69 mm, giving a surface area of ​​1. The construction of the substrate and the attachment of components to the substrate are described in more detail below.

[0107] The top side 171 of the substrate 170 supports, for example, eight DRAMs 110a, which may be arranged in a 2x4 grid extending from one edge 170a to the opposite edge 170b of the substrate. The 2x4 grid of DRAMs 110a is then arranged approximately equidistant between two other edges 170c, 170d, effectively forming a strip along the center of the module 100.

[0108] Additionally, the underside 172 of the substrate supports eight DRAMs 110b. The DRAMs 110b on the underside 172 of the substrate are positioned in positions corresponding to the DRAMs 110a on the top side 171. In other words, each DRAM 110a on the top side 171 is positioned directly above a DRAM 110b on the bottom side 172.

[0109] It will be appreciated that "upper side" and "lower side" as used herein are merely labels referring to two sides of substrate 170, and that module 100 may be mounted during use such that lower side 172 is not below upper side 171.

[0110] Each DRAM 110 may be a DDR (Double Data Rate) DRAM. In one example, each DRAM is a LPDDR (Low Power DDR) DRAM (e.g., an LPDDR5 DRAM). Each DRAM may have a capacity of 16 GB, although in other examples, the capacity may be 24 GB or 32 GB. LPDDR DRAM is designed for mobile computing situations (e.g., mobile phones or laptop computers). However, the inventors have found that such memory can advantageously provide high-capacity, low-latency memory suitable for meeting the demands of artificial intelligence / machine learning models in high-performance computing situations.

[0111] Module 100 further includes four fabric chips 140 mounted on top side 171. Fabric chips 140 are sometimes referred to herein as "routing chips" or "memory mounting and routing chips" given their above-described function of routing data between different processor cores 20 and between processor cores 20 and DRAM 110. Fabric chips 140 are positioned in areas 170e or 170f between substrate edge 170c or 170d and strips of DRAM 110.

[0112] Each fabric chip 140 is adjacent to a different pair of DRAMs 110a on top side 171 and, consequently, an additional pair of DRAMs 110b on bottom side 172. A fabric chip 140 is connected to its four adjacent DRAMs 110. In one example, a fabric chip 140 is connected to only its four adjacent DRAMs 110.

[0113] Thus, module 100 can be divided into four conceptual quadrants by a first conceptual line 170y extending between the centers of edges 170a and 170b and a second conceptual line 170x extending therebetween, with each quadrant including a fabric chip 140 and four DRAMs 110a, 110b connected to fabric chip 140. The module is mirror-symmetric about both lines 170x and 170y. One quadrant 102q of module 100 is shown in Figure 12. Each quadrant 102q can be considered a sub-module of module 100.

[0114] Module 100 includes two connection components 160 disposed on underside 172 of substrate 170. One connection component 160 is positioned below region 170e, and the other connection component 160 is positioned below region 170f, such that each connection component 160 underlies two fabric chips 140. Each connection component 160 is configured to mate with a corresponding connection component (420, see FIG. 18 ) formed on another substrate (e.g., motherboard 400). Thus, module 100 is connectable to and separable from motherboard 400 via connection components 160. Thus, connection components 160 form electrical couplings or links between module 100 and the rest of the system outside module 100.

[0115] Module 100, and more particularly, each fabric chip 140, connects to processor core 20 via connection components 160. Each connection component provides a number of connectors, as described below. Each fabric chip 140 connects via one or more connectors of connection components 160 disposed thereon. The connectors of connection components 160 can therefore be considered part of the physical embodiment of links L1-L4 described above with respect to FIGS. 5 and 6, in that the links are part of the signal paths extending between processor core 20 and fabric chip 140.

[0116] Additionally, the connectors of the connectivity components 160 provide part of the physical implementation of the links between each module 100 and other modules 100 in other pods, and to the rest of the system. Thus, the modules 100 do not include processor cores 20, but instead provide routing for data and memory access between the processor cores 20. In other words, the only processing power for the module 100 is that provided in the fabric chips 140. The processor cores 20 are located remotely from the module 100 and do not form part of the module 100.

[0117] Furthermore, as mentioned above, there are no high-bandwidth direct connections between fabric chips 140. Thus, each fabric chip 140 on a module 100 is not connected to other fabric chips 140 on the same module 100.

[0118] Each connection component 160 may take the form of a mezzanine connector. The connection component 160 may be, for example, a hermaphroditic mezzanine connector with 11 rows, each row having 15 pairs of pins, sometimes referred to as a connector. An example pair of pins 161 is labeled in FIG. 13 , with the remaining pins not labeled for clarity. The mezzanine connector may be a Mirror Mezzanine connector supplied by Molex®. In other examples, other connection components 160 may be used. For example, connectors supplied by Samtec®, TE Connectivity®, or Amephenol® may be used. The connection component 160 may be part of a physical linkage between the module 100 and the motherboard, providing physical support for the module 100. A more detailed description of the pins 161 is provided below with reference to FIG. 16 .

[0119] The structure of the substrate 170 and the connections between the elements of the module 100 and the substrate 170 will now be further described.

[0120] Substrate 170 is a package substrate. As such, substrate 170 is not a traditional printed circuit board, but instead is a type of substrate typically used in chip packages to support a chip die. Substrate 170 is sometimes referred to as a high-density interconnect (HDI) substrate or an interposer substrate. The use of the term "interposer" in this context does not imply that the substrate serves as an intermediate or intervening layer; instead, the substrate is merely a reference to the type of substrate used. As will be apparent from the description herein, package substrate 170 is the primary substrate of module 100 and does not serve as an interposer.

[0121] In one example, the package substrate 170 is a High Tg glass epoxy multilayer material (eg, MCL-E-705G provided by Hitachi®).

[0122] In one example, substrate 170 is monolithic, i.e., a single complete substrate. In other examples, substrate 170 may include two or more substrates coupled together physically, electrically, or physically and electrically.

[0123] As shown in Figure 14a, the package substrate 170 includes a core 173 and a plurality of accumulation layers 174 formed on the core 173. The core 173 has two layers 173a and 173b. The first core layer 173a is insulating and provides strength to the substrate. The second layer 173b may be a copper layer. The thickness of the core 173 may be approximately 1.2 mm.

[0124] The accumulation layers 174, shown in more detail in FIG. 14b, each have a plurality of conductive lines or wires 177 that electrically connect the elements of the module 100. The accumulation layers 174 may each include copper foil sub-layers 174a that form the conductive lines and insulating sub-layers 174b that insulate each accumulation layer 174 from the other accumulation layers 174. The thickness of each copper foil sub-layer 174a may be approximately 12 microns. The thickness of each insulating sub-layer 174b may be approximately 30 microns. It should be understood, therefore, that FIG. 14 is not to scale and emphasizes the size of the accumulation layers 174 relative to the core 173.

[0125] In one example, six accumulation layers 174 are formed on each side of core 173 to provide a 6:2:6 package substrate. The accumulation layers 174 on one side of substrate 170 may each have a different function. For example, one or more of the layers 174 may be ground layers including conductive lines 177 connected to ground. One or more of the layers may be VDD layers including conductive lines 177 connected to VDD. One or more of the layers 174 may be signal layers that transmit signals between connection components 160 and fabric chip 140 and between fabric chip 140 and DRAM 110. In one example, two of the accumulation layers 174 are signal layers. The outermost layer 174 may include pads (not shown) for connection to other elements of module 170.

[0126] 14a, vias 174c may be formed between accumulation layers 174 so that conductive lines 177 may pass between layers 174. Additionally, core vias 175 may be formed through core 173 so that conductive lines 177 may pass from the top side to the bottom side of substrate 170.

[0127] Fabric chip 140 is a flip chip that is directly attached to substrate 170. In other words, fabric chip 140 is a semiconductor chip that is manufactured to include solder bumps on the face of the die. These bumps are then attached directly to substrate 170. In one example, the bump pitch in the core region of the chip is approximately: x=261 microns y=154 microns Diagonal pitch = 151 microns where x is the width direction between the long edges of the chip and y is the length direction between the short edges of the chip. The pitch may be larger to provide connections to the DRAM 110, e.g., x=286 microns, y=164 microns, diagonal pitch=167 microns. The structure and function of fabric chip 140 are described in more detail below with reference to FIG. 15.

[0128] The conductive lines 177 of the package substrate 170 are sufficiently thin in the area of ​​the substrate 170 beneath each fabric chip 140 to allow the lines to break out from the chip 140 footprint.

[0129] The DRAMs 110 are mounted to the substrate 170 using a ball grid array (BGA). That is, the DRAMs 110 each take the form of a packaged semiconductor chip including a die and a package substrate. The die is secured to the top side of the package substrate and electrically connected to the package substrate. The package substrate has a grid of solder balls formed on the underside of the package substrate, which are then secured to corresponding conductive pads on the substrate 170. The BGA may have a pitch of, for example, 650 microns. Thus, the ball pitch is substantially coarser than the bumps on the fabric chip 140.

[0130] The connection components 160 may be connected to the substrate 170 via a BGA. Thus, each connection component 160 may include a grid of solder balls arranged on a face of the connection component 160 opposite the face including the pins 161.

[0131] Figure 15 illustrates in more detail the structure of fabric chip 140. Bumps formed on the underside of fabric chip 140 for connection to a substrate are illustrated as small circles generally designated by the numeral 141. Blocks 142, 143, and 144 shown in Figure 15 illustrate areas within fabric chip 140 where specific circuitry may be placed.

[0132] The die of fabric chip 140 is rectangular and has two opposing long edges 140a, 140b and two opposing short edges 140c, 140d. The short edges 140c, 140d are approximately 6 mm long. The long edges 140a, 140b are approximately 15 mm long. In one example, each fabric chip is 5.5 mm by 15.3 mm long. Thus, the aspect ratio of chip 140 is approximately 3:1. In one example, fabric chip 140 is a single or monolithic die.

[0133] Fabric chip 140 includes multiple memory controllers 142a-142h. Each memory controller 142 is a circuit formed on the die of the chip that serves as an interface to DRAM 100. In an example where the DRAM is an LPDDR DRAM, memory controller 142 is an LPDDR interface. The LPDDR interface meets the relevant LPDDR standard (e.g., JEDEC standard (e.g., JESD209-5B)). In an example where DRAM 110 is a different type of DRAM, memory controller 142 may meet the appropriate standard required to access DRAM 110.

[0134] As described above, each fabric chip 140 is associated with two DRAMs 110 on the upper side and two DRAMs 110 on the lower side 172. Each of these DRAMs 110 may have multiple memory channels (e.g., four memory channels). Thus, each DRAM 110 may be a four-channel DRAM. The fabric chip 140 includes a number of memory controllers 142 sufficient to access the channels of the DRAMs 110. In the illustrated example, the fabric chip 140 includes eight memory controllers 142a-142h, each of which is a dual-channel memory controller 142. This provides the 16 channels required to access four four-channel DRAMs. In one example, the memory channels are each 16 bits wide.

[0135] 15, the memory controller 142 is located on one long side 140a of the chip 140. Each chip 140 is placed in the module 100 so that this side 140a faces the DRAM 100. This placement can reduce breakout of connections to the DRAM 100 from under the chip 140 footprint.

[0136] The memory controllers 142 may be arranged in a 2x4 grid, with the first four of memory controllers 142a-142d positioned closest to the long side 140a and the second four of memory controllers 142e-142h positioned inboard of the first four memory controllers 142a-142d. The first four of memory controllers 142a-142d may communicate with DRAM 110 on the top side 171 of the substrate 110. The second four memory controllers 142e-142h may be positioned to access DRAM 110 on the bottom side 172 through core vias 175 formed under the fabric chip 140. This arrangement also helps to mitigate breakout from underneath the footprint of the chip 140.

[0137] A plurality of bumps 141 of the fabric chip 140 are disposed under each memory controller 142 (at least some of which are memory attachment connectors) that are in electrical communication with the DRAM 110 via conductive traces 177 in the substrate 170. In other words, the bumps 141 under each memory controller include memory attachment ports connectable to the DRAM 110. Other bumps under each memory controller 142 may be connected to the memory controller's power supply or ground.

[0138] Each memory controller 142 forms part of the above-mentioned DDR interface block 48. Accordingly, the routing logic of the fabric chip 140 is configured to route data to and from the DRAM 110 through the memory controller 142.

[0139] The fabric chip 140 further includes a plurality of link controllers 143, 144. Each link controller 143, 144 may include circuitry formed on a die. A first group of link controllers 143 each includes four communication lanes, labeled 143-1 through 143-4. Only one of the link controllers 143 is labeled in this manner to improve clarity of the drawing.

[0140] Each communication lane 143-1 through 143-4 forms an individual communication link to an external device (i.e., a device not on module 100). Accordingly, at least some of the bumps 141 under each lane 143-1 through 143-4 of link controller 143 form an external link port or connector. The external link port is connected to connection component 160 via conductive lines 177. Bumps 141 may include a bump for a transmit signal and a bump for a receive signal.

[0141] Each of the communication lanes 143-1 to 143-4 may be a serial link, such as a SERDES link as described above. Accordingly, the link controller 143 may include analog circuitry. In one example, the link controller 143 provides a 100 Gbps link.

[0142] In addition to the four communication lanes 143-1 to 143-4, the link controller 143 may include a common area 143-5 that provides common functions to the four lanes 143-1 to 143-4 (eg, a common clock signal for each communication lane).

[0143] Three of the link controllers (labeled 143a-c) provide communication lanes for communication with each processor core 20. Thus, each fabric chip 140 has 12 communication lanes for communication with the processor core 20. As described above with respect to FIG. 9, three links (e.g., L2a, L2b, L2c) are provided to each processor core. Thus, each communication lane corresponds to an EPC (e.g., EPC2a, EPC2b, EPC2c) shown in FIG. 9. Thus, the 12 communication lanes on the fabric chip 140 provide connections to four processor cores 20.

[0144] An additional link controller (labeled 143d) provides four additional communication lanes. This link controller may include three lanes for pod-to-pod communication. The communication lanes may implement EPCs corresponding to Pa, Pb, and Pc in FIG. 9 to implement pod-to-pod links PLa, PLb, and PLc. The pod-to-pod links PL are sometimes referred to as cluster connecting links, in that the pod-to-pod links connect fabric chips to different clusters.

[0145] The link controller 143d may include lanes for system-to-system communication, where one of the communication lanes implements the EPC (i.e., corresponding to the PCS in FIG. 9) to provide a system link SL, which connects to, for example, a switching fabric.

[0146] Link controller 144 provides a PCIe link to the host computer. Link controller 144 may include two sub-controllers (not shown), each implementing four lanes of communication. Link controller 144 may provide a lower speed connection compared to link controller 143.

[0147] Link controllers 143, 144 may be located along the long edge 140b opposite memory controller 142. Again, this helps to mitigate breakout of the wires from underneath fabric chip 140.

[0148] In one example, link controller 144 can only operate on one of the four fabric chips 140 on module 100. The link controllers 144 of the other three fabric chips 140 may not be connected to connection component 160 and therefore cannot communicate with the host.

[0149] The remaining bumps 141 of the fabric chip 140 that are not under one of the controllers 142, 143, 144 may include main chip power supply bumps and ground bumps. The main chip power supply may be a different power supply than the power supply of the memory controller 142. Additionally, some of the bumps 141 (e.g., bumps at the corners of the chip 140) may be dummy bumps that are not electrically connected. These dummy bumps are most susceptible to the different thermal expansion characteristics of the substrate 170 and the chip 140 and therefore cannot be used to reliably transmit signals.

[0150] Figure 16 shows the two connection components 160a, 160b in more detail, with the other components of the module 100 omitted for clarity, apart from a box representing the position of the fabric chip 140 relative to the connection component 160.

[0151] Approximately half of the pairs of pins 161 are VSS or ground pins. Thus, in each group of pins described below, approximately half of the pairs of pins perform the associated function and approximately half serve as VSS.

[0152] Connection component 160 includes groups of pins 181 that carry signals to and from link controller 143. These pins 181 may therefore function as a SERDES link. One group of pins 181 carries signals to and from a particular fabric chip 140. In particular, pin 181-1 carries signals to and from fabric chip 140-1, pin 181-2 carries signals to and from fabric chip 140-2, pin 181-3 carries signals to and from fabric chip 140-3, and pin 181-4 carries signals to and from fabric chip 140-4. The pins that form the SERDES links are generally located toward the outer edges of connection component 160 (i.e., on the side of the connection component farthest from DRAM 110).

[0153] Each group of pins 181 may include a transmit pin 181a and a receive pin 181b configured for transmitting and receiving signals, respectively. To maintain clarity of the drawing, only a selection of the transmit pin 181a and receive pin 181b of pin 181-1 is labeled. Other groups of pins 181 include similarly arranged transmit pins 181a and receive pins 181b. In the example shown in FIG. 16, the transmit pins 181a and receive pins 181b are interspersed. In other words, the transmit pins 181a and receive pins 181b are distributed throughout group 181, such that a transmit pin 181a may be located adjacent to (i.e., next to) a receive pin 181b. Adjacent or adjacent in this context includes pairs of pins that are diagonally arranged relative to one another.

[0154] Additionally, connection component 160 includes groups of pins 182 that carry power supplies for the fabric chips. In one example, the power supplies carried by pins 182 are the main digital power supplies for fabric chips 140. One group of pins 182 supplies a particular fabric chip 140 with the same group and subscript (i.e., -1, -2, etc.). Power supply pins 182 are typically located below the fabric chip 140 that they supply.

[0155] Connection component 160 includes a group of pins 183 in communication with link controller 144, which carries the PCIe link to the host computer. As mentioned above, only one of the fabric chips (e.g., fabric chip 140-1) may have an associated link controller 144. Thus, only one of connection components 160a may include PCI link pins 183. Pins 183 are generally located toward the center of connection component 160a.

[0156] Connection component 160 further includes clock pins 184-1 through 184-4 that carry clock signals for link controller 143 of each fabric chip 140. Connection component 160a may also include clock pin 185-1 for the PCIe controller. Because only one fabric chip 140 has a PCIe link enabled, only one clock pin 185-1 may be provided.

[0157] Additionally, connection component 160 may include pin 186 that provides a power supply to link controller 143. In particular, pin 186 provides power to the PHY or analog components of link controller 143. Link controller 143 (in particular the PHY) may require a power supply that is different from the main digital power supply. For example, the link controller may require a quieter power supply. Additionally, connection component 160 may include pin 187 that provides a power supply to DRAM 110.

[0158] 19 shows another example pair of connection components 260a, 260b. Connection component 260 operates similarly to connection component 160, with corresponding elements having corresponding reference numerals increased by 100. For clarity, only the differences will be detailed below.

[0159] Connection component 260 includes groups of pins 281 that transmit signals to and from the link controller, corresponding in function to pins 181 described above. In contrast to pins 181, in each group of pins 281, transmit pins 281a are separated from receive pins 281b; that is, transmit pins 281a are not located adjacent to receive pins 281b. In this example, adjacent includes diagonally adjacent pins.

[0160] For example, each group 281 may include an area including transmit pins 281a and an area including receive pins 281b. In other words, transmit pins 281a and receive pins 281b each form a continuous area. The two areas 281a and 281b are separated from each other by a group of ground pins 281c. The group of ground pins 281c forms a buffer or separation area between transmit pins 281a and receive pins 281b.

[0161] Furthermore, the transmit pins 281a associated with one fabric chip 240 are positioned so that they are not adjacent to the transmit pins 281a associated with another fabric chip 240. For example, the transmit pins 281a of group 281-1 associated with fabric chip 240-1 are positioned at or toward one end of the connection component 260, while the transmit pins 281a of group 281-2 associated with fabric chip 240-2 are positioned at or toward the opposite end of the connection component 260. However, the receive pins 281b of the two groups 281-1, 281-2 may be positioned adjacent to each other.

[0162] Advantageously, isolating transmit pin 281a from receive pin 281b has been found to reduce or prevent near-end crosstalk (NEXT) between transmit pin 281a and receive pin 281b. NEXT occurs due to the relative difference in signal strength between the transmit and receive signals, resulting from signal degradation and attenuation as the signal travels between the transmit and receive points. Thus, the signal transmitted via transmit pin 281a has significantly higher gain than the signal received via receive pin 281b, thus causing crosstalk.

[0163] Additionally, connection components 260a, 260b differ from connection components 160a, 160b in that they use pins located along the long edge of connection component 260 closest to DRAM 110 as DRAM power supply pins 287. It is understood that features of connection components 160 and 260 may be combined or interchanged.

[0164] 18 shows an example of a power supply arrangement for module 100. As shown in FIG. 18, a motherboard 400 onto which module 100 can be mounted includes a power supply component 400P. Power supply component 400P includes, for example, a load power supply point located on the opposite side of module 100 from motherboard 400. Power is supplied from power supply component 400P to fabric chip 140 via connection components 420 and 160.

[0165] Thus, the module 100 may not include a power supply (e.g., point of load power delivery). This allows the substrate 170 to be smaller, reducing the use of relatively high-cost substrate material. Furthermore, positioning the fabric chips 140 on the opposite side of the substrate 170 directly above the connection components 160 minimizes the distance between the power delivery components 400P and the fabric chips 140, reducing IR drop.

[0166] FIG. 17 illustrates a method for manufacturing an example module 100.

[0167] The method includes providing a substrate (S171). As described above, the substrate may be a package substrate 170 having multiple layers 174. The substrate may be formed by repeatedly arranging copper foil underlayers 174a and insulating underlayers 174b on a core 173. In some examples, layers 174 may be provided on both sides of the core 173. In some examples, vias 174c are formed between layers and / or vias 175 are formed through the core 173, for example, by laser drilling.

[0168] In step S172, conductive lines 177 are formed on the substrate 170. In one example, the lines 177 are formed by etching the copper foil underlayer 174a. However, other methods may be used to form the conductive lines 177. The conductive lines 177 may pass through vias 174c, 175.

[0169] In step S173, a first semiconductor chip is attached directly to the substrate 170 by flip-chip attachment. The first chip may be a fabric chip 140. Solder bumps formed on the first chip may be attached to corresponding pads formed on the substrate 170 by heating the module 100. The module 100 may be heated by passing the module 100 through a suitable oven. Each of multiple first chips (e.g., four fabric chips 140) may be attached simultaneously, for example, in the same pass through an oven or other heating cycle.

[0170] In some examples, an underfill material (e.g., epoxy resin) is underfilled onto the first chip. The underfill material may be supplied as a liquid by capillary action caused by a narrow gap formed between the underside of the chip and the substrate, which draws the liquid between the solder bumps. The underfill may then be cured by a further heating cycle that is cooler than the heating cycle used to melt the solder bumps. The underfill helps redistribute stresses caused by the different thermal expansion coefficients of the chip and the substrate.

[0171] In step S174, a packaged semiconductor chip having a BGA is attached to the substrate. The packaged semiconductor chip may be a DRAM 110 described herein. For example, the module 100 may be heated by passing the module 100 through a suitable oven to attach the solder balls of the BGA of the packaged chip to corresponding pads formed on the substrate. Heating the module 100 to attach the packaged semiconductor chip may be a separate heating cycle from the heating cycle to attach the first semiconductor chip. Underfilling the first semiconductor chip can ensure that the first chip remains attached to the substrate during subsequent heating of the module to attach the packaged semiconductor chip.

[0172] Step S174 may include mounting a packaged semiconductor chip on one side of the substrate (e.g., top side 171) and then mounting a packaged semiconductor chip on the other side of the substrate (e.g., bottom side 172). Thus, there may be two separate heating cycles for mounting BGAs on each side of the substrate. In other examples, BGAs on both sides of substrate 170 may be mounted in a single heating cycle.

[0173] In step S175, the connection components 160 are attached to the substrate. In one example, the connection components 160 have BGAs. For example, the module 100 may be heated by passing it through an appropriate oven, thereby attaching the solder balls of the BGAs of the connection components to corresponding pads formed on the substrate. In some examples, the connection components 160 are attached to the substrate in an additional heating cycle following the attachment of the packaged semiconductor chips. In other examples, the connection components 160 may be attached to the substrate in the same heating cycle as one or more of the packaged semiconductor chips. For example, the connection components 160 may be attached to the underside 172 in the same heating cycle that attaches the packaged semiconductor chips to the substrate 170.

[0174] In some instances, the order of steps may be changed: the packaged semiconductor chip and / or the connecting components 160 may be secured to the substrate before flip-chip attaching the first chip.

[0175] Various modifications may be made to the module 100 described above. In some examples, the number of fabric chips 140 and DRAMs 110 present in the module 100 may differ from the examples described above. For example, each fabric chip 140 may be connected to fewer DRAMs 110 (i.e., one, two, or three DRAMs) or more DRAMs 110 (five or more DRAMs, e.g., eight DRAMs). In other examples, the module 100 may include fewer fabric chips 140 or more fabric chips 140. The module 100 may include fewer of the conceptual quadrants described above (e.g., two quadrants, six quadrants, eight quadrants, or any other suitable number). In some examples, the number of connection components 160 provided may vary. For example, only one connection component 160 or more than two connection components 160 may be provided. Furthermore, the elements of the fabric chip 140 may vary. For example, fabric chip 140 may include more or fewer memory controllers and / or link controllers that communicate with a different number of DRAMs 110, processor cores 20, and other quads or pods. The locations of memory controllers 142 and link controllers 143, 144 may vary on fabric chip 140. The bumps on fabric chip 140 may have different functions and / or be arranged differently. The pins 161 of connection components 160 may be arranged differently and / or have different functions.

[0176] Advantageously, module 100 provides routing capabilities and large-capacity, high-bandwidth, low-latency memory for processor cores 20, making it suitable for processing large-scale machine learning models. When arranged as shown in FIG. 8 with a ratio of two modules 100 to each processor core 20, each processor can access up to 512 GB with an example bandwidth of 9.6 Tbit / s. Furthermore, the use of module 100 limits the links required for processor cores 20, conserving beachfront space normally used for memory access and routing.

[0177] Advantageously, module 100 includes direct flip-chip attached fabric chip 140 and DRAM 110 and connection components 160 attached to module 100 via BGA. By flip-chip attaching fabric chip 140 directly to the substrate, fabric chip 140 does not require additional packaging, thus reducing the overall size of module 100.

Claims

1. a memory and routing module, A substrate; a connection component mounted on the substrate, the connection component including a plurality of pins for mating the memory and routing module with a corresponding connection component on a motherboard; a dynamic random access memory (DRAM) chip disposed on the substrate; a routing chip mounted on the substrate, a memory controller; a plurality of connections including a first group of connections between the memory controller and the DRAM chips and a second group of connections with the pins of the connection component; routing logic configured to route data between the second group of connections and the first group of connections; a routing chip including: Including, the second group of connections includes a plurality of processor connections each configured to attach to a respective processor chip; the routing logic is configured to route a signal from one of the processor connections to another of the processor connections; Memory and Routing Module.

2. 10. The memory and routing module of claim 1, wherein the DRAM chip forms part of a DRAM chip package that includes a ball grid array (BGA) that connects the DRAM chip package to the substrate.

3. 10. The memory and routing module of claim 1, wherein the routing chip includes a plurality of bumps that secure the routing chip directly to the substrate by flip-chip attachment.

4. 2. The memory and routing module of claim 1, further comprising a plurality of DRAM chips disposed on the substrate, the routing chip including a plurality of memory controllers, each memory controller connected to a respective DRAM chip of the plurality of DRAM chips via the plurality of connections.

5. 5. The memory and routing module of claim 4, further comprising four DRAM chips connected to said routing chip.

6. 5. The memory and routing module of claim 4, wherein at least one of the plurality of DRAM chips is disposed on a first side of the substrate and at least one of the plurality of DRAM chips is disposed on a second side of the substrate.

7. 10. The memory and routing module of claim 1, comprising a plurality of routing chips and a plurality of DRAM chips disposed on the substrate, each routing chip including a memory controller connected to each DRAM chip.

8. The memory and routing module of claim 1 , wherein the connection component is a mezzanine connection component.

9. the routing chip is disposed on a first side of the substrate; the connection components are mounted on the second side of the substrate at locations corresponding to the locations of the routing chips; 10. The memory and routing module of claim 1, wherein the routing chip is configured to receive power via the connection component from a power supply component electrically coupled to the connection component.

10. The memory and routing module of claim 9 , including a plurality of vias in the substrate that form electrical paths connecting the connection components to the routing chip.

11. 2. The memory and routing module of claim 1, wherein the DRAM chips are low power double data rate (LPDDR) memory and the memory controller includes a LPDDR interface.

12. 2. The memory and routing module of claim 1, wherein the memory controller is located on a long edge of a die of the routing chip, the long edge of the die facing the DRAM chip.

13. The memory and routing module of claim 1 , wherein the processor connection is a serializer / deserializer (SERDES) connection.

14. the plurality of pins including processor connection pins configured to support the plurality of processor connections; the processor connection pins include a transmit pin and a receive pin; The receiving pins are arranged so as not to be adjacent to the transmitting pins.

10. The memory and routing module of claim 1.

15. 2. The memory and routing module of claim 1, wherein the substrate is a package substrate including a plurality of conductive lines, and the first group of connections and the second group of connections are via the plurality of conductive lines.

16. A system comprising the memory and routing module of any one of claims 1 to 15 and a plurality of processor chips connected to the memory and routing module via connection components.

17. Each processor chip does not communicate directly electrically with other processor chips, Each processor chip is configured to communicate with the other processor chips via a routing chip; 17. The system of claim 16.

18. 17. The system of claim 16, wherein the plurality of processor chips perform time-deterministic processing.

Images