DRAM MODULE WITH DATA ROUTING LOGIC

Abstract

The memory and routing module (100) includes a substrate (170) and a connection component (160). The connection component (160) is mounted to the substrate (170) and includes a number of pins (161) that connect the module (100) to a corresponding connection component (160) on the motherboard (400). The substrate (170) is connected to a dynamic random access memory (DRAM) chip (110) and a routing chip (140). The routing chip (140) includes a memory controller (142), a number of connections, and routing logic (46). The number of connections includes a first group between the memory controller (142) and the DRAM chip (110) and a second group of connections with the pins (161) of the connection component (160). The routing logic (46) routes data between the second group of connections and the first group of connections.

Description

[Technical field]

[0001] The present disclosure relates to a memory and routing module and a system including the module. [Background technology]

[0002] The demand for high performance computing continues to grow, especially in efforts to meet the demands of artificial intelligence / machine learning models that impose challenging requirements on computational resources. It is known to address this particular demand by interconnecting multiple processing chips in clusters, where the processing chips are configured to work collaboratively to meet the demands of processing power needed to process large scale AI / ML models.

[0003] Another requirement of high performance computing is the ability to access large amounts of memory. To increase memory capacity, attempts are being made to connect external memories to processing nodes in clusters. Such external memories may be connected by memory buses that provide an access path between the external memories and the processing nodes. These memory buses may take the form of parallel or serial links. For example, dynamic random access memories (DRAMs) may be mounted in dual in-line memory modules (DIMMs) on server racks. These can provide scalable memory capacity on the order of terabytes. Such DIMMs may be mounted vertically on the server racks and many DIMMs can be stacked together to provide the required memory capacity for a computer. Summary of the Invention [Means for solving the problem]

[0004] The inventors seek to address the coupling problem by providing a cluster of processing chips that have access to large memory and also allow the processing chips to communicate with each other to increase processing power for specific tasks.

[0005] The present inventors have recognized certain shortcomings with the current nature of cluster connectivity. Silicon chips contain circuits arranged in a two-dimensional manner on a die surface that is typically divided into "cores" (e.g., processor cores 2) surrounded by a perimeter or "beachfront" 7 (see FIG. 1). The beachfront is used for input / output (IO) circuits located on the edge of the chip to reduce signal breakout to packaged pins. For example, the beachfront 7 is used to accommodate the inter-processor links 6a, 6b and the processor-memory links 8a...8d to external memory illustrated in FIGS. 2 and 3.

[0006] The area of ​​the beachfront depends on the type and bandwidth of the I / O requirements. High performance computing chips often use an approximate maximum manufacturable die size ("full scale") of about 25.5 mm by 32.5 mm, requiring a beachfront depth of about 2 mm on each of the four edges of the die. With current lithography techniques, a full scale die results in a die core of about 21.5 mm by about 28.5 mm, which is about 74% of the total die area. The computational resources of the die are constrained to this core portion, and the inventors have found that the cost of the beachfront area is quite high. Figure 1 shows an example of a die with a full perimeter beachfront (all four edges) labeled processor cores 2 and 7.

[0007] The present disclosure may help address these problems, as well as any other problems that will be apparent to the skilled reader from the disclosure contained herein.

[0008] According to a first aspect of the disclosure, a memory and routing module, A substrate; a connection component mounted on the substrate, the connection component including a number of pins for mating the module with a corresponding connection component on a motherboard; a dynamic random access memory (DRAM) chip disposed on the substrate; A routing chip mounted on a substrate, A memory controller; a plurality of connections including a first group of connections between the memory controller and the DRAM chips and a second group of connections having pins of a connection component; routing logic configured to route data between the second group of connections and the first group of connections; a routing chip including: The present invention provides a memory and routing module including:

[0009] The DRAM chip may form part of a DRAM chip package that includes a ball grid array (BGA) that connects the DRAM chip package to a substrate.

[0010] The routing chip may include a number of bumps that secure the routing chip directly to the substrate by flip-chip attachment.

[0011] The memory and routing module may include a number of DRAM chips disposed on a substrate. The routing chip may include a number of memory controllers. Each memory controller may be connected to each DRAM chip of the number of DRAM chips via a number of connections. The module may include four DRAM chips connected to the routing chip.

[0012] At least one of the plurality of DRAM chips may be disposed on a first side of the substrate, and at least one of the plurality of DRAM chips is disposed on a second side of the substrate.

[0013] The memory and routing module may include a number of routing chips and a number of DRAM chips arranged on a substrate, with each routing chip including a memory controller connected to each DRAM chip.

[0014] The connection component may be a mezzanine connection component. The connection component may be hermaphroditic. The module may include a plurality of connection components. The module may include a pair of connection components. The connection component may be a ball grid array mounted connection component.

[0015] The routing chip may be mounted and positioned on a first side of the substrate. The connection components may be mounted on a second side of the substrate at locations corresponding to the locations of the routing chip. The routing chip may be configured to receive power from a power supply component electrically coupled to the connection component, for example, on a motherboard, via the connection components. The module may include a plurality of vias forming an electrical path connecting the connection components to the routing chip to the substrate. The module may not include a power supply component (e.g., a load power supply point). The routing chip may receive power only via the connection components. The plurality of pins may include a plurality of routing chip power supply pins. The routing chip power supply pins may be configured to receive power from the power supply component. The power supply pins may be positioned on the second side of the substrate at locations corresponding to the locations of the routing chip. In examples including a plurality of routing chips, the connection component may include a plurality of groups of power supply pins, each group configured to receive power from the power supply component and provide the received power to a respective one of the plurality of routing chips. Each group of power supply pins may be located on the second side of the substrate at a location corresponding to the location of each routing chip.

[0016] The DRAM chips may be low power double data rate (LPDDR) memory. The memory controller may include a LPDDR interface.

[0017] The memory controller may be located on a long edge of the routing chip's die, which may face the DRAM chips.

[0018] The second group of connections may include a plurality of processor connections, each configured to mount to a respective processor chip. The routing logic may be configured to route a signal from one of the processor connections to another of the processor connections. The processor connections may be serializer / deserializer (SERDES) connections.

[0019] The plurality of pins may include processor connection pins configured to support a plurality of processor connections. The processor connection pins may include transmit and receive pins. The receive pins may be positioned so as not to be adjacent to the transmit pins. In this context, "adjacent" may mean pins adjacent to a given pin, including diagonally. At least a portion of the receive pins may be separated from the transmit pins by one or more ground pins.

[0020] In examples including multiple routing chips, the multiple pins may include multiple groups of processor connection pins, each group configured to support multiple processor connections of one of the multiple routing chips. Each group of processor connection pins may include a transmit pin and a receive pin. The receive pins in the group may be arranged so as not to be adjacent to the transmit pins in the group. At least some of the receive pins in the group may be separated from the transmit pins in the group by one or more ground pins. A transmit pin of one of the multiple groups of processor connection pins may be arranged so as not to be adjacent to another receive pin of the multiple groups of processor connection pins. Suitably, the transmit pins of each group are arranged so as not to be adjacent to the receive pins of any other of the multiple groups.

[0021] The plurality of pins may include a DRAM chip power supply pin configured to receive power for the DRAM chip. The DRAM chip power supply pin may be disposed along an edge of the connection component. The edge may be an edge of the connection component closest to the DRAM chip.

[0022] The substrate may be a package substrate including a plurality of conductive lines. The first group of connections and the second group of connections may be via the plurality of conductive lines.

[0023] According to a second aspect of the disclosure, there is provided a system including a memory and routing module as described herein and a number of processor chips connected to the memory and routing module via connection components.

[0024] The processor chips may not be in direct electrical communication with each other: each processor chip may be configured to communicate with the other processor chips through a routing chip.

[0025] Multiple processor chips may perform time-deterministic processing.

[0026] The system may include a motherboard to which the modules can be attached. The motherboard may be configured for attachment of multiple processor chips. The system may include multiple modules. The motherboard may include power supply components that provide power to the modules.

[0027] The disclosure also extends to methods of routing data using the memory and routing modules described herein.

[0028] Further optional features of the system and corresponding methods of the second aspect are described above with respect to the modules of the first aspect and may be combined in any combination.

[0029] According to a fourth aspect of the disclosure, a package substrate for housing a flip-chip mounted semiconductor chip in a module; a first flip-chip mounted semiconductor chip mounted on a package substrate; a first ball grid array mounted packaged semiconductor chip mounted on a package substrate, the first flip chip mounted semiconductor chip and the first ball grid array mounted semiconductor chip being in electrical communication with each other; a connection component mounted on the package substrate, the connection component including an electrical coupling for coupling the package substrate to a corresponding connection component on a motherboard; Including, the package substrate includes a first ball grid array mounted semiconductor chip mounted on the package substrate and a plurality of conductive lines coupling the first flip chip mounted semiconductor chip to the connecting components; Provide a module.

[0030] The first ball grid array mounted semiconductor chip may be a dynamic random access memory (DRAM) chip. The DRAM chip may be a LPDDR chip. The module may include a plurality of ball grid array mounted semiconductor chips.

[0031] The package substrate may be a monolithic package substrate, and at least some of the plurality of ball grid array attached packaged semiconductor chips may be disposed on the monolithic package substrate.

[0032] The module may include a plurality of flip chip mounted semiconductor chips mounted to a package substrate. The plurality of flip chip mounted semiconductor chips may be in electrical communication with the plurality of ball grid array mounted semiconductor chips. Each flip chip mounted semiconductor chip may be in electrical communication with a subset of the plurality of ball grid array mounted semiconductor chips. Each flip chip mounted semiconductor chip may be in electrical communication with four ball grid array mounted semiconductor chips. The module may include four flip chip mounted semiconductor chips.

[0033] The first flip-chip mounted semiconductor chip may include routing logic configured to route data between connection components mounted to the package substrate and the first ball grid array mounted packaged semiconductor chip.

[0034] The conductive traces coupling the first flip-chip mounted semiconductor chip to the connection component may include a plurality of processor connections. The first flip-chip mounted semiconductor chip may include routing logic configured to route data from one of the processor connections to another of the processor connections. The processor connections may include serial connections, such as serializer / deserializer (SERDES) links.

[0035] The first flip chip mounted semiconductor chip may be mounted to a first side of the package substrate.The first ball grid array mounted semiconductor chip may be mounted to the first side of the package substrate.The first ball grid array mounted semiconductor chip may be mounted to a second side of the package substrate.

[0036] The module may include a second ball grid array mounted semiconductor chip mounted on a second side of the package substrate. The package substrate may include a plurality of vias forming electrical paths electrically connecting the second ball grid array mounted semiconductor chip to the first flip chip mounted semiconductor chip. At least one of the vias may be disposed under the first flip chip mounted semiconductor chip.

[0037] The first flip-chip mounted semiconductor chip may be mounted on a first side of the package substrate, and the connection component may be mounted on a second side of the substrate at a location corresponding to the location of the first flip-chip mounted semiconductor chip. The first flip-chip mounted semiconductor chip may be configured to receive power from a power supply component electrically coupled to the connection component via the connection component. The module may include a plurality of vias forming an electrical path connecting the connection component to the package substrate and the first flip-chip mounted semiconductor chip. The module may not include a power supply component (e.g., a load power supply point). The first flip-chip mounted semiconductor chip may receive power only via the connection component.

[0038] The first side and the second side of the substrate may be opposing sides of the substrate.

[0039] The package substrate may include a plurality of layers suitably formed on a core. At least two of the layers may include conductive traces for transmitting signals between the first flip-chip mounted semiconductor chip and the first ball grid array mounted packaged semiconductor chip, and between the first flip-chip mounted semiconductor chip and a connecting component. The substrate may be a high density interconnect (HDI) substrate.

[0040] The connection component may be a mezzanine connector. The connection component may include a plurality of pins. The connection component may be hermaphroditic. The module may include a plurality of connection components. The module may include a pair of connection components. The connection component may be a ball grid array mounted connection component.

[0041] According to a fifth aspect of the disclosure, there is provided a system including a module as described herein and a plurality of processor chips connected to the module via a connection component.

[0042] According to a sixth aspect of the disclosure, there is provided a system including a module as described herein and a motherboard to which the module is mountable. The motherboard may be configured for mounting a processor chip. The system may include the processor chip. The system may include multiple processor chips and multiple modules. The motherboard may include power supply components that provide power to the module.

[0043] According to a seventh aspect of the disclosure, 1. A method of manufacturing a module, comprising the steps of: providing a package substrate; forming a plurality of conductive traces on a package substrate; mounting the first semiconductor chip to a package substrate by flip-chip mounting; attaching a ball grid array packaged semiconductor chip to a package substrate; attaching connection components to the package substrate, the connection components including electrical connections that couple the package substrate to corresponding connection components on a motherboard; Including, a plurality of conductive lines electrically connecting the first semiconductor chip to the ball grid array packaged semiconductor chip and the connecting components; A method is provided.

[0044] The method may include heating the module and mounting the first semiconductor chip prior to mounting the ball grid array packaged semiconductor chip or connecting components. The method may include underfilling the first semiconductor chip.

[0045] The method may include the steps of heating the module and mounting the ball grid array packaged semiconductor chips prior to mounting the connecting components, and heating the module and mounting the connecting components.

[0046] The method may include forming a plurality of vias in a package substrate, mounting a first semiconductor chip to a first side of the package substrate, and mounting at least one of the ball grid array packaged semiconductor chip or a connecting component to a second side of the package substrate. At least one of the conductive lines may pass through the vias and connect the first chip to the ball grid array packaged semiconductor chip or the connecting component.

[0047] The method may include forming a plurality of layers on a package substrate core, at least two of the layers including conductive lines transmitting signals between a first semiconductor chip and a ball grid array packaged semiconductor chip and connecting components.

[0048] Further optional features of the method of the seventh aspect are described above in relation to the modules of the fourth aspect and may be combined in any combination.

[0049] Furthermore, the features of the modules, systems, and methods described in the first to third aspects may be combined with the modules, systems, and methods of the fourth to seventh aspects.

[0050] In certain embodiments of the present invention, any processor chip may access any memory attached to any of the fabric chips in the computer cluster. Memory access may be via high speed serial links. Additionally, any processor may exchange packets with any other processor in the computer via routing logic in the fabric chips.

[0051] In certain aspects of the present invention, the inventors enable clustering of processing chips in multiple tiers.

[0052] In certain aspects of the present invention, each processing chip itself improves processor core area for a particular size of substrate.

[0053] Another requirement imposed by high performance computing is the ability to have a high bandwidth to large capacity memories. Currently, so-called high bandwidth memories (HBM) are implemented by providing the memory within the physical structure of the processing node itself, i.e., in close proximity to the processing chips that are mounted on a silicon substrate in the package that forms the processing node. In effect, the HBM is butted up against the processing chips on the silicon substrate, so as to be as physically close as possible to the processing chips that provide the processing functionality. In this way, although high bandwidth is achieved, there is a limit to the memory capacity based on the physical size of the memory that can be accommodated in this type of structure. Furthermore, such HBMs are expensive to manufacture.

[0054] In the field of artificial intelligence (AI) and machine learning (ML), mathematical models can be extremely large and require extremely large amounts of memory to accommodate the mathematical models. As the size of the models increases, the cost of providing HBM also increases.

[0055] Currently, the lack of availability of large capacity high bandwidth memory creates constraints on the size and nature of models available to machine learning / artificial intelligence computers. In particular, the knowledge capacity of a model is a function of the amount of memory that is reasonably accessible. In some embodiments of the invention, a portion of the beachfront may be no longer used for connection to external memory and made available to the HBM.

[0056] For a better understanding of the present invention and to show how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]

[0057] [Figure 1] FIG. 1 is a simplified block diagram of a chip connected to a memory. [Diagram 2] 1 is a schematic diagram of multiple interconnected chips. [Diagram 3] FIG. 1 is a block diagram of multiple processor chips connected using a switch core. [Figure 4]1 is a simplified block diagram of a processor chip with a reduced beachfront. [Diagram 5] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips. [Figure 6] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips having a higher ratio of fabric chips to processor chips. [Figure 7] 1 is a simplified block diagram of a computer including interconnected processor chips and fabric chips, each processor chip with high bandwidth memory. [Figure 8] 1 is a simplified block diagram of a computer including a set of interconnected clusters. [Figure 9] FIG. 2 is a simplified block diagram of a fabric chip. [Figure 10] 1 is a schematic diagram of an example processor chip. [Figure 11a] FIG. 2 is a top perspective view of an example memory and routing module. [Figure 11b] FIG. 11b is a bottom perspective view of the example memory and routing module of FIG. [Figure 12] 12 is a schematic diagram of a top view of the example memory and routing module of FIG. 11. [Figure 13] 13 is a schematic diagram of the underside of the example memory and routing module of FIGS. 11 and 12. FIG. [Figure 14a-14b] FIG. 14 is a schematic diagram of the example memory and routing module of FIGS. 11-13. [Figure 15] 1 is a schematic diagram of an example layout of a fabric chip. [Figure 16] 14 is a schematic diagram illustrating pins of connection components of the example memory and routing modules of FIGS. 11-13. FIG. [Figure 17] 1 is a simplified flowchart of an example method for manufacturing a memory and routing module. [Figure 18] 14 is a simplified cross-sectional view of the example memory and routing module and motherboard of FIGS. 11-13. [Figure 19]14 is a schematic diagram illustrating pins of another example of a connection component pair usable in the memory and routing modules of FIGS. 11-13. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0058] In the drawings, corresponding reference characters indicate corresponding components. Those skilled in the art will appreciate that the elements in the drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the drawings may be exaggerated relative to other elements to help facilitate a better understanding of the various example embodiments. Moreover, to facilitate easy viewing of the illustrations of these various example embodiments, common well-understood elements that are useful or necessary in commercially viable embodiments are often not shown.

[0059] There are various known methods for forming clusters of processing chips by interconnecting the processing chips with each other.

[0060] Figure 1 shows an example of a processor chip intended to be connected in a processing cluster. The processing chip 1 includes a processor core 2 (shown cross-hatched) implemented on a silicon die 4. It is convenient to distinguish between a beachfront area where external links are provided and a core area for the processing circuitry of the processor core. The beachfront area includes inter-processor links 6a, 6b, and processor-memory links 8a, 8b, 8c, 8d (shown in Figure 1) connected to respective DRAMs 10a, 10b, 10c, 10d.

[0061] Figure 2 illustrates a cluster of four processing units of the type illustrated in Figure 1 with exhaustive direct connectivity of inter-processor links. Each processor core 2a, 2b, 2c, 2d is shown with three external connectors 5 along the top edge and three external connectors 5' along the bottom edge. In the cluster of Figure 2, each processor core is connected to each other processor core by two external connection links attached to the external connectors in an exemplary manner. See, for example, links L and L' connecting processor core 2a to processor core 2d. Note that there remains a need for a dedicated processor chip-memory bus to attach DRAM.

[0062] This is just one example of inter-processor connectivity in a cluster.

[0063] An alternative way of connecting processor chips together in a cluster is to use a switch fabric. Figure 3 is a schematic diagram illustrating the connection of four processor cores 2a, 2b, 2c, 2d to each of two switch cores 12a, 12b. Each switch core can be made to route traffic between specific processor cores under program control. In this arrangement, each processor has access to its own externally connected DRAM.

[0064] In the above examples, each processing chip has access to memory. In some previous examples, that memory may be externally attached memory connected to each processor core of the cluster, and / or high bandwidth memory (HBM) connected within the processor package. In either case, the memory attachment uses the "beach front" of the die.

[0065] In a particular embodiment of the present disclosure, a computer includes a plurality of processor chips and fabric chips arranged in a cluster. Within the cluster, each processor chip is connected to all fabric chips, and each fabric chip is connected to all processor chips in an exhaustive bipartite connection configuration. There are no direct connections between the fabric chips themselves in the cluster. Furthermore, there are no direct connections between the processor chips themselves. Each fabric chip has routing logic configured to route incoming packets from one processor chip connected to the fabric chip to another processor chip. Furthermore, each fabric chip has a means for attaching to an external memory. The routing logic can route packets between a processor connected to the fabric chip and a memory attached to the fabric chip. The fabric chip itself includes a memory controller that performs memory control functions that manage memory accesses from and to the memory attached to the fabric chip.

[0066] In certain embodiments described further herein, clusters of processing chips and fabric chips may themselves be interconnected to form a larger computer system. Each processor chip in the cluster may access any of the memory attached to any of the fabric chips in the cluster. This greatly expands the memory capacity available to any particular processor chip.

[0067] The connection configurations described herein have the further advantage that, in certain embodiments, it is not necessary to use the entire edge of the processor die to surface the external connections.

[0068] The inventors have found that by limiting the beachfront required for connections to less than all four edges of the die, it is advantageous to free up more silicon for fabrication of the processing "cores." For example, if only the short edges of a full reticle die are used for I / O, the area available for the processor cores on the chip increases to about 88% of the total die area, about 19% more than for the four-sided case. Figure 4 shows an example of such a chip 1' where the vertical edges cannot accommodate the beachfront, and the top and bottom edges each have a beachfront 7a, 7b.

[0069] Prior art processing cluster connectivity requirements include a beachfront around the entire perimeter (e.g., as shown in FIG. 1). Certain implementations of the present connectivity configuration described herein allow for the use of processor dies that have beachfronts only on their top and bottom edges, but not on their vertical edges (as shown in FIG. 4).

[0070] In accordance with the presently described embodiment of the invention, multiple processors are connected in a cluster using one or more "fabric chips." Each fabric chip provides access to external memory (e.g., DRAM) and also provides routing for inter-processor traffic. Reference is now made to FIG. 5, which illustrates four processor chips 20a, 20b, 20c, 20d. Each processor chip includes a processor core area 22a, 22b, 22c, 22d that extends to each vertical edge of the chip. Each processor chip has an upper beachfront area 30a and a lower beachfront area 30b (only chip 20a is shown). Upper beachfront area 30a has a set of external port connections C1, C2, C3, C4 (labeled only for processor chip 20a). Each processor chip can also be seen to have four external port connections in the upper beachfront area. Similarly, the lower beachfront area of ​​each processor chip has four external port connections labeled C5, C6, C7, C8. Note that the bottom set of external port connections is labeled only for processor chip 20a. It will be appreciated that the other processor chips similarly each have a set of external port connections in their bottom beachfront regions.

[0071] The cluster of FIG. 5 further includes eight "fabric chips." Each fabric chip includes a fabric core 40a, 40b, . . . 40h. Each fabric chip has a lower beachfront area 44a, . . . 44h with a set of external ports. These external ports are provided only on the fabric chip 40a at the port connections labeled FC1, FC2, FC3, FC4. It can be seen that each fabric chip has a corresponding set of external ports on its respective lower beachfront area. On the upper beachfront area of ​​each fabric chip, one or more memory attachment interfaces are provided that allow the fabric chip to connect to one or more memories, exemplified in FIG. 5 as respective DRAMs 10a, 10b, 10c, 10d, . . . to 10p. For example, the fabric core 40a shown in FIG. 5 is connected to two DRAMs 10a, 10b by suitable memory attachment interfaces provided on the upper beachfront 46a of the fabric chip. Other large capacity memories may be connected, such as double data rate DRAM (DDR), and more recent manifestations of that DRAM, such as low power DDR (LPDDR). The high bandwidth connections between the processor chips and the fabric chips in the cluster are "exhaustively bifurcated." This means that each processor chip is connected to every fabric chip, and each fabric chip is connected to every processor chip. The connections are via links (e.g., L1) between the processor ports at the port connections (e.g., C1) and the fabric chip ports at the port connections (e.g., FC1). However, it should be noted that in the illustrated example, there are no direct high bandwidth connections between the processor chips or between the fabric chips in the cluster. Furthermore, in the illustrated example, there is no externally attached memory directly connected to each processor (although there may be high bandwidth memory within the chip package (see below)). Each fabric chip provides a routing function that provides routes between every pair of processors, and between each processor and the memory attached to the fabric chip.

[0072] Furthermore, the links may be manifested in any suitable manner. Each link may be connected or reconnected to a different port to set up the computer configuration. Once the computer configuration is set up and operational, the links are not multiplexable and do not fan in or fan out. That is, instead, there are no intermediate switches that directly connect a port on the processor to an end port on the fabric chip. Any packet sent on the link is received at a port at the other end of the fixed link. The link is advantageously bidirectional, and preferably the link can operate in both directions at the same time, although bidirectionality is not a mandatory requirement. One particular category of communication links is the SERDES link, which has power requirements that are independent of the amount of data transmitted on the link, or the time consumed transmitting that data. SERDES is an acronym for Serializer / Deserializer, and such links are known. For example, a SERDES link may be implemented using twisted pair wires. To transmit a signal on the wires of such a link requires power to be applied to the wires, and the voltage is changed to generate the signal. SERDES links have the property that there is a fixed power for the bandwidth capacity on the SERDES link, whether it is used or not. This is due to the need to provide clocking information on the link by constantly switching the current or voltage state of the line, even when no data is being transmitted. As is known, data is transmitted by holding the state of the line and indicating a logical "0" or a logical "1". SERDES links are implemented at each end by circuitry that connects the link layer devices to the physical link (e.g., a copper wire). This circuitry is sometimes called the PHY (physical layer). In this example, layers 1 and 2 of the Ethernet protocol are used to transmit packets over the link. However, it will be appreciated that any data transmission protocol may be used.

[0073] There are several advantages to the computer described herein.

[0074] There is no longer a need to dedicate a fixed percentage of the processor beachfront (and therefore I / O bandwidth) to a fixed amount of memory or inter-processor connections: all processor I / O bandwidth passes through the fabric chip and is available on demand for either purpose (memory or inter-processor).

[0075] Under some popular models of microprocessor computation, such as massively synchronous parallel (BSP), the peak DRAM bandwidth and peak inter-processor bandwidth usage may not be simultaneous. Thus, the total bandwidth requirement can be met with a smaller processor beachfront, giving the processor chip a larger core area. BSP is known per se in the art. According to BSP, each processing node performs a computation phase and an exchange phase (also called a communication or message passing phase) in alternating cycles. The computation phase and the exchange phase are performed by the processing chip that executes the instructions. During the computation phase, each processing unit performs one or more computation tasks locally, but does not communicate any results of these computations to other processing chips in the cluster. In the exchange phase, each processing chip may exchange one or more processing results from a previous computation phase to one or more other processing chips in the cluster. It should be noted that different processing chips may be assigned to different groups for synchronization purposes. According to the principles of BSP, a barrier synchronization is performed at the junction where the computation phase transitions to the exchange phase, or where the exchange phase transitions to the computation phase, or at both junctions. That is, require all processing chips in a group to complete each computation step before any of the group can proceed to the next switching step, or require all processing chips in a group to complete each switching step before any processing chip in the group can proceed to the next computation step, or implement both of these conditions. This sequence of switching and computation steps is repeated over a number of cycles. In BSP terminology, each iterative cycle of switching and computation steps is sometimes called a "superstep."

[0076] This has the practical effect that there are situations where there is no simultaneous use of all links required to access memory (for the purposes of the computation phase) and the links used to exchange data between processing chips in the switching phase. As a result, there is maximum efficient utilization of fixed links without compromising memory access times and inter-processor switching delays. Nevertheless, it will be appreciated that the embodiments described herein have applications other than when used with BSP or other similar synchronization protocols.

[0077] Links can be dynamically deactivated while not in use so as to effectively not consume power. However, the activation times and non-deterministic nature of machine learning applications generally make dynamic activation during program execution problematic. As a result, the inventors have determined that it is better to use the fact that link power consumption is essentially constant for any particular configuration, and therefore the best optimization is to maximize the use of physical links by maintaining as much parallel inter-processor and processor-memory activity as possible.

[0078] All memory in the cluster is accessible to each processor without having to go around in a loop through another processor. This shared memory arrangement can benefit software efficiency.

[0079] In the example shown in FIG. 5, there are two "ranks" of fabric chips, one mounted on each top and bottom edge of the processor chip. The top rank includes fabric cores 40a...40d, connected to each processor core by a respective link. For example, processor core 20a is connected to fabric core 40a by link L1, to fabric core 40b by link L2, to fabric core 40c by link L3, and to fabric core 40d by link L4. The bottom rank includes fabric cores 40e...40h. Fabric core 40a is also connected to each processor core 20a...20d by corresponding links (not labeled, although shown in the drawing, for reasons of clarity). There is no use of vertical processing chip edges for beachfronts.

[0080] However, there are different design choices within the overarching concept: for example, the vertical edge of the processor can be used to provide more bandwidth to the fabric chips, and all links coming out of the beach front of the processor chip can be passed to one rank or three ranks of the fabric chips, etc.

[0081] The number of fabric chips in each rank may differ from the number of processor chips. What remains important to achieving the advantages of the present invention is maintaining full bipartite connectivity between the processing chips and the fabric chips, with routing functions and external memory access provided by the fabric chips.

[0082] FIG. 6 shows a specific example of connecting four processing chips to eight fabric chips in an upper rank and eight fabric chips in a lower rank. Each processing chip is connected to sixteen fabric chips. Take the processor chip 20a for example. The processor chip has eight upper rank connectors C1, C1', C2, C2', C3, C3', C4, C4', each connected to a respective link connector on each fabric core, labeled 40a, 40a' in FIG. 6, with two fabric cores. The processor chip has eight lower rank connectors C5, C5', C6, C6', C7, C7', C8, C8', each connected to a respective link connector on each of the eight fabric chips in the lower rank. Each fabric chip is connected to four processor chips.

[0083] It should be noted that the use of external connectors to provide exhaustive bipartite connectivity in a cluster according to examples of the invention does not preclude the presence of other I / O ports on the processor chip or fabric chip. For example, a particular one of the processor chips or fabric chips in a cluster may be provided with an I / O port that allows connectivity between multiple clusters, or to a host device, etc. In one embodiment described with reference to Figures 8 and 9, the fabric chip provides this additional connectivity.

[0084] It should further be noted that additional memory can be mounted directly on the processor chip, for example along a vertical edge. That is, additional high bandwidth memory (HBM) can be provided in close proximity to the processing chip mounted on the silicon substrate in the package forming the processing node. In fact, the HBM is butted against the processing chip on the silicon substrate so as to be as physically close as possible to the processing chip providing the processing function. For example, the high bandwidth memory (HBM) can be mounted on the processor chip, while the large capacity memory can be mounted on the fabric chip, combining the advantages of both memory types in a cluster. FIG. 7 illustrates an embodiment in which high bandwidth memory (HBM) modules 26 are mounted on the east and west edges of each processor chip 20'a, 20'b, 20'c, 20'd. In other respects, the computer illustrated in FIG. 7 has the same connections as the computer described in FIG. 5. The HBM 26 can be mounted by short parallel connections of memory buses formed on the substrate, or by using silicon bridges on the package substrate.

[0085] In the exemplary computer described herein, the processor chip 20 is not intended to be deployed standalone. Instead, the deployment of the processor chip is in a computer cluster where the processor chip is supported by one or more fabric chips 40. The processor chips 20 connect to each other via the fabric chips 40, allowing the use of all processor chip links L1, L2, etc. for simultaneous use as inter-processor links and memory access links. In this manner, the computer provides a larger capacity, faster memory system than existing computer systems. In current computer systems, providing large capacity, high bandwidth memory is becoming increasingly expensive. Furthermore, limitations remain on the processing power that can be obtained while providing high bandwidth memory access and large capacity memory. The present computer can overcome those limitations.

[0086] By providing the routing logic in the fabric chips, the processor chips do not need to have routing logic for external routing functions, which frees up silicon area to maximize I / O bandwidth per processor chip and also maximizes the area available for processing circuitry within the processor core.

[0087] By placing link ports along the north and south edges, this frees up the east / west edges so that processor cores can run to the east / west edges, thus maximizing processing power, or the east / west edges can be left free for high bandwidth memory consolidation.

[0088] A computer may be operated in different topologies. In one example, a group of four processor chips and eight fabric chips (e.g., as illustrated in FIG. 5) may form a cluster. Within a cluster, each group of four fabric chips connected to one of the processor chip edges is referred to herein as a rank. A cluster of five contains two ranks.

[0089] A pod may contain multiple clusters. Clusters may be interconnected within a pod using processor-to-processor links in the fabric chips. Pods may be interconnected with each other using pod-to-pod links in the fabric chips. These are shown in more detail in FIG. 9, which illustrates an example fabric chip.

[0090] FIG. 8 is a schematic diagram of a system topology and hierarchy according to one embodiment. FIG. 8 illustrates a number of pods P1, P2, P3, . . . Pn (labeled POD16). In the example of FIG. 8, n=8, but it is readily apparent that a different number of pods can be connected to a computer system using the techniques described herein. One of the pods, pod P1, is shown in detail. Pod P1 includes four clusters Q1, Q2, Q3, Q4. In the example of FIG. 8, each cluster includes four processor chips 20a, 20b, 20c, 20d that share 32 fabric chips. A fabric chip 40 is labeled in FIG. 8, with Q4 indicating, for example, that fabric chip 40 (Q4) is in cluster Q4. As shown in FIG. 8, in each cluster Q1, Q2, Q3, Q4, the four processor chips 20a, 20b, 20c, 20d are connected to the 32 fabric chips in an exhaustive bisection arrangement. That is, as described above, each fabric chip in a cluster is connected to a total of four processor chips, and each processor chip is connected to a total of 32 fabric chips. Each processor chip has 32 port connections C1, C2, ... C32 (16 on the top edge and 16 on the bottom edge). As illustrated in Figure 9, in a particular embodiment, each port connection provides three bidirectional serial links, for a total of 96 processor links. Each set of 12 processor external links (out of the 96 links) connects to each set of four fabric chips (three processor links to each fabric chip port connection FC). Thus, eight sets of 12 links connect to eight sets of four fabric chips within a cluster. Four clusters Q1, Q2, Q3, Q4 are grouped together to form a pod, with pod-to-pod links of 32 fabric chips within each cluster. Each cluster sends three bundles to each of the 32 links, and each bundle connects to each of the other three clusters. A bundle of pod-facing links between two clusters includes one link between each of the 32 corresponding peer fabric chips in the two clusters. A particular pod-facing link may be connected to a third-party Ethernet switch.

[0091] FIG. 9 is a simplified block diagram of components on the fabric chip 40. As shown in FIG. 9, routing logic 46 is connected to the DDR interface block 48 for transferring data packets between the DDR interface block 48 and other ports. Further, routing logic 46 is attached to each processor-facing link port. Each port includes an Ethernet port controller EPC. Routing logic is attached to the Ethernet port controller of the pod-facing port and the Ethernet port controller of the system-facing link. Further, routing logic 46 is attached to the PCI complex for interfacing to the host system. PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high-speed computers.

[0092] FIG. 9 shows an example of a fabric chip that can be used to configure a computer by connecting computer clusters together in a hierarchical manner, in addition to inter-processor and processor-to-memory communication. First, the components of the fabric chip used to perform inter-processor and processor-to-memory communication are described. Each fabric core port connection includes three serial links. Each serial link includes a port with an Ethernet port controller (EPC). As described, these links may be SERDES links, e.g., twisted pair wires that enable serial packet communication.

[0093] For reasons of clarity, not all components in FIG. 9 are illustrated with relevant reference. Each fabric core connection FC1, FC2, FC3, and FC4 has a configuration as described herein with reference to a fabric core port connection FC2 that connects to a second processor (e.g., processor 20b in FIG. 6). The fabric connection FC2 includes three links L2a, L2b, L2c, each including an Ethernet port controller EPC2a, EPC2b, EPC2c, respectively. It should be noted that in other embodiments, a single physical link may be provided, or a different number of physical links may be provided for each fabric chip connection FC. Thus, it should be noted that the link labeled L2 in the previous figures may include three individual serial links (e.g., L2a, L2b, and L2c). The routing logic 46 in the fabric chip 40 may be implemented as a ring router, a crossbar router, or in any other manner. Additionally, the fabric chips are connected to external memories (e.g., DRAMs 10A, 10B, etc.) (not shown in FIG. 9). While two DRAMs are shown in the previous figures, in the embodiment of FIG. 9 the fabric chip is connected to four DRAMs. To make this connection the fabric chip includes four DRAM interface blocks DIB1, DIB2, DIB3 and DIB4 associated with four DDR sub-connect layers DDR sub1, DDR sub2, DDR sub3 and DDR sub4 respectively. Each DDR interface block DIB 48 incorporates a memory controller which manages access to the memory attached to the block. One memory attachment interface 44 is shown in FIG. 9, however it will be appreciated that each DDR sub-layer has a respective memory attachment interface which attaches to an external DRAM. The routing logic 46 is configured to route memory access packets received from the attached processor core to an addressed one of the data interface blocks DIB1-DIB4. Additionally, the routing logic 46 is configured to route from one attached processor chip to another attached processor chip via each fabric chip port.In certain embodiments, the routing logic prevents memory packets (e.g., memory access response packets) from being routed from one memory attached interface to another. In such embodiments, the memory response packets may simply be routed to the processor chip via the correct port attached to the routing logic 46. For example, an incoming packet on link L2a of fabric core port connection FC2 is routed to an addressed port connected to the routing logic 46 based on the packet's routing information. For example, if the packet is intended to be routed to processor 20c, the routing logic 46 identifies processor 20c from the packet's routing information and causes the packet to exit via an Ethernet port controller onto a link attached to processor 20c.

[0094] If the packet is a memory access packet, the routing logic routes the packet to the appropriate DDR interface block based on the memory address of the packet. Note that in this embodiment, each DDR interface block DIB1...DIB4 includes four memory access channels. It is understood that any number of memory access channels can be provided by each interface block DIB1...DIB4. The memory access channels are managed by a memory controller in each data interface block DIB1...DIB4.

[0095] As mentioned above, in the example shown in FIG. 9, the fabric chip 40 has additional components that allow the computer to be configured in interconnected clusters. To this end, the fabric chip includes a pod-facing port connection PL. The pod-facing port connection PL includes three ports, each with an Ethernet port controller Pa, Pb, Pc connected to each link. The routing logic detects a packet whose packet information indicates that the packet should not be routed to a processor in this cluster, but instead to a processor in another cluster, and routes the packet to one of the pod-facing ports. It should be noted that the pod-facing port connection PL can send packets to a corresponding pod-facing port in a fabric chip on another cluster, or receive packets from a corresponding pod-facing port on a fabric chip of another cluster.

[0096] The fabric chip of FIG. 9 may also route the packet to another pod in the system. To this end, a system port SL is provided. The system port includes a corresponding Ethernet port controller EPC and is connected to a system serial link connected to a corresponding port in another pod. The routing logic may determine that the packet is intended to be routed to another pod in the system and send the packet to the system port SL. The packet may be received on the system port SL from a corresponding system port of another fabric chip in another pod in the system connected via the system serial link and may be applied to the routing logic.

[0097] It will be appreciated that any type of routing logic may be utilized to route traffic from one external connection of a fabric chip to another connection of the fabric chip to another processor chip via an external port or to an attached memory via a memory attached interface. The term data packet as used herein means a string of bits that includes a payload to be transmitted between processor chips or between a processor chip and a memory attached to a fabric chip. The packet includes information (e.g., a destination identifier and / or memory address for routing). In some embodiments, the destination processor identifier may be included in the packet header. One type of ring routing logic is described in Graphcore's UK Patent Application No. 2115929.8.

[0098] As described herein, each processing chip can perform a processing or computational function. There are many possible different manifestations of suitable processing chips. Graphcore has developed an Intelligent Processing Unit (IPU) as described, for example, in U.S. Patent Application Nos. 15 / 886009, 15 / 886053, and 15 / 886131 [PWF Refs. 408525US, 408526US, and 408527US], the contents of which are incorporated herein by reference. FIG. 10 is a schematic diagram of an IPU. The IPU includes multiple tiles 103 on a silicon die, each including a processing unit with local memory. The tiles communicate with each other using time-deterministic exchanges. Each tile 103 has an instruction store that holds a local program, an execution unit that executes the local program, a data store that holds data, an input interface with a set of input lines, and an output interface with a set of output lines. A switching fabric 101 (sometimes called a switch or switching fabric) is connected to each tile by a respective set of output lines and to each tile by a respective set of input lines via switching circuits connectable by each tile. A synchronization module (not shown) is operable to generate synchronization signals and to switch between the computation phase and the switching phase. The tiles execute their local programs in the computation phase according to a common clock that may be generated by the die or received by the die. At a given time in the switching phase, the tiles execute send instructions from their local programs to send data packets on an output set of connection lines, the data packets being destined for at least one recipient tile but not having a destination identifier that identifies that recipient tile. At a given switch time, the recipient tile executes switch control instructions from its local program to control the switching circuits to connect an input set of lines to the switching fabric and receive the data packets at a receive time. The send times at which data packets are destined to be sent from the sending tile and the given switch times are controlled by the common clock for the synchronization signal.

[0099] Time-deterministic exchanges allow efficient transfers between tiles on a die. Each tile has its own local memory that provides data storage and instruction storage. The IPU is further connected to external memory that can transfer data to the IPU for use by the tile via the fabric chip as described herein.

[0100] The tiles 103 of the IPU may be programmed such that data packets sent by a SEND instruction from a local program are intended to access memory (memory access packets) or have a destination of another IPU connected in the cluster or system. In such a case, the data packets are sent by the originating tile 103 to the switching fabric but are not picked up by the recipient tile in the IPU. Instead, the switching fabric ensures that the tiles are equipped with the appropriate connectors C1, C2, etc. for external communication from the IPU. Packets for off-chip communication are generated to include information defining the final off-chip destination but not the external port to which it should be sent. Packets may be sent to an external port using the principle of time-deterministic switching to identify the external port for the packet when code is compiled against the tile. For example, memory access packets may identify a memory address. Packets for another IPU may include an identifier of the other IPU. This information is used by routing logic on the fabric chip to correctly route off-chip packets generated by the IPU.

[0101] The diagram in Figure 10 shows five example regions of an example IPU chip separated by four dashed boundary lines 105. Note that the dashed lines represent abstract boundaries 105 of abstract regions on the processor chip shown for illustrative purposes, and that the boundaries 105 do not necessarily represent physical boundaries on the IPU chip.

[0102] From the top to the bottom of the diagram in FIG. 10, the regions separated by boundary line 105 are the upper beach front, the upper tile region, the switching fabric region, the lower tile region, and the lower beach front, respectively.

[0103] The above presents a logical arrangement of the computer system described herein, including processor cores or chips 20, fabric chips 40 and DRAM 10. Below, the physical layout and organization of some elements of the computer system are described in more detail.

[0104] 11a-14b, a memory and routing module 100 is shown according to an example of the disclosure.

[0105] Module 100 includes multiple fabric chips 140, multiple DRAMs 110, and two connection components 160. Fabric chips 140 and DRAMs 110 correspond to fabric chips 40 and DRAMs 10 described above. That is, fabric chips 140 and DRAMs 110 described below may incorporate the features described above in terms of fabric chips 40 and DRAMs 10. Fabric chips 140 on module 100 include a memory controller, described in more detail below, that accesses DRAMs 110.

[0106] The fabric chips 140, the DRAM 110 and the connection components 160 are mounted on a substrate 170 in the form of a planar plate. The plate has dimensions of approximately 80 mm by 70 mm, for example, approximately 5300-5400 mm. 2 The substrate may be 77 mm by 69 mm, giving a surface area of ​​100 mm. The construction of the substrate and the attachment of components to the substrate are described in more detail below.

[0107] The top side 171 of the substrate 170 supports, for example, eight DRAMs 110a, which may be arranged in a 2x4 grid extending from one edge 170a to the opposing edge 170b of the substrate. The 2x4 grid of DRAMs 110a is then arranged approximately equidistant between two other edges 170c, 170d, effectively forming a strip along the center of the module 100.

[0108] Additionally, the underside 172 of the substrate supports eight DRAMs 110b. The DRAMs 110b on the underside 172 of the substrate are positioned in positions corresponding to the DRAMs 110a on the topside 171. In other words, each DRAM 110a on the topside 171 is positioned directly above a DRAM 110b on the bottomside 172.

[0109] It will be appreciated that "upper side" and "lower side" as used herein are merely labels referring to the two sides of substrate 170, and that module 100 may be mounted during use such that lower side 172 is not below upper side 171.

[0110] Each DRAM 110 may be a DDR (Double Data Rate) DRAM. In one example, each DRAM is a LPDDR (Low Power DDR) DRAM (e.g., LPDDR5 DRAM). Each DRAM may have a capacity of 16 GB, but in other examples, the capacity may be 24 GB or 32 GB. LPDDR DRAM is designed for mobile computing situations (e.g., mobile phones or laptop computers). However, the inventors have found that such memory can advantageously provide a large capacity low latency memory suitable for meeting the demands of artificial intelligence / machine learning models in high performance computing situations.

[0111] Additionally, module 100 includes four fabric chips 140 mounted on top side 171. Fabric chips 140 are sometimes referred to herein as "routing chips" or "memory mounting and routing chips" given their above-mentioned function of routing data between different processor cores 20 and between processor cores 20 and DRAM 110. Fabric chips 140 are positioned in areas 170e or 170f between edges 170c or 170d of the substrate and strips of DRAM 110.

[0112] Each fabric chip 140 is adjacent to a different pair of DRAMs 110a on top side 171, and consequently an additional pair of DRAMs 110b on bottom side 172. Fabric chip 140 is connected to those four adjacent DRAMs 110. In one example, fabric chip 140 is connected to only those four adjacent DRAMs 110.

[0113] Thus, the module 100 can be divided into four conceptual quadrants by a first conceptual line 170y extending between the centers of the edges 170a and 170b and a second conceptual line 170x extending therebetween, each quadrant including a fabric chip 140 and four DRAMs 110a, 110b connected to the fabric chip 140. The module is mirror symmetrical about both lines 170x and 170y. One quadrant 102q of the module 100 is shown in FIG. 12. Each quadrant 102q can be considered a sub-module of the module 100.

[0114] The module 100 includes two connection components 160 disposed on the underside 172 of the substrate 170. One connection component 160 is positioned under an area 170e, and the other connection component 160 is positioned under an area 170f, such that each connection component 160 is below two fabric chips 140. Each connection component 160 is configured to mate with a corresponding connection component (420, see FIG. 18) formed on another substrate (e.g., a motherboard 400). Thus, the module 100 is connectable to and separable from the motherboard 400 by the connection components 160. Thus, the connection components 160 form an electrical coupling or link between the module 100 and the rest of the system outside the module 100.

[0115] Module 100, and more particularly each fabric chip 140, connects to processor core 20 via connection component 160. Each connection component provides a number of connectors as described below. Each fabric chip 140 connects via one or more connectors of connection component 160 disposed thereon. The connectors of connection component 160 are therefore considered part of the physical embodiment of links L1-L4 discussed above with respect to Figures 5 and 6, in that the links are part of the signal paths extending between processor core 20 and fabric chip 140.

[0116] Additionally, the connectors of the connectivity components 160 provide part of the physical implementation of the links between each module 100 and other modules 100 in other pods, and to the rest of the system. Thus, the modules 100 do not include processor cores 20, but instead provide routing for data and memory access between the processor cores 20. In other words, the only processing power for the module 100 is that provided in the fabric chips 140. The processor cores 20 are located remotely from the module 100 and do not form part of the module 100.

[0117] Additionally, as mentioned above, there are no high bandwidth direct connections between fabric chips 140. Thus, each fabric chip 140 on a module 100 is not connected to other fabric chips 140 on the same module 100.

[0118] Each connection component 160 may take the form of a mezzanine connector. The connection component 160 may be, for example, a hermaphroditic mezzanine connector with eleven rows, each row having fifteen pairs of pins, sometimes referred to as connectors. An example pair of pins 161 is labeled in FIG. 13, the remaining pins are not labeled for clarity. The mezzanine connector may be a Mirror Mezz connector supplied by Molex®. In other examples, other connection components 160 may be used. For example, connectors supplied by Samtec®, TE Connectivity® or Amephenol® may be used. The connection component 160 may be part of a physical linkage between the module 100 and the motherboard, providing physical support to the module 100. A more detailed description of the pins 161 is provided below with respect to FIG. 16.

[0119] The structure of the substrate 170 and the connections between the elements of the module 100 and the substrate 170 will now be further described.

[0120] Substrate 170 is a package substrate. As such, substrate 170 is not a traditional printed circuit board, but instead is a type of substrate typically used in chip packages to support a chip die. Substrate 170 may also be referred to as a high density interconnect (HDI) substrate or an interposer substrate. Use of the term "interposer" in this context does not imply that the substrate serves as an intermediate or intervening layer, but instead is merely a reference to the type of substrate that is used. As will be apparent from the description herein, package substrate 170 is the primary substrate of module 100 and does not serve as an interposer.

[0121] In one example, the package substrate 170 is a High Tg glass epoxy multi-layer material (eg, MCL-E-705G provided by Hitachi®).

[0122] In one example, substrate 170 is monolithic, i.e., a single complete substrate, while in other examples, substrate 170 may include two or more substrates coupled together physically, electrically, or physically and electrically.

[0123] As shown in Fig. 14a, the package substrate 170 includes a core 173 and a number of accumulation layers 174 formed on the core 173. The core 173 has two layers 173a, 173b, where the first core layer 173a is insulating and serves to provide strength to the substrate. The second layer 173b may be a copper layer. The thickness of the core 173 may be about 1.2 mm.

[0124] The accumulation layers 174, shown in more detail in FIG. 14b, each have a plurality of conductive lines or wires 177 that electrically connect the elements of the module 100. The accumulation layers 174 may each include a copper foil sub-layer 174a that forms the conductive lines, and an insulating sub-layer 174b that insulates each accumulation layer 174 from the other accumulation layers 174. The thickness of each copper foil sub-layer 174a may be approximately 12 microns. The thickness of each insulating sub-layer 174b may be approximately 30 microns. It should be understood, therefore, that FIG. 14 is not to scale and emphasizes the size of the accumulation layers 174 relative to the core 173.

[0125] In one example, six accumulation layers 174 are formed on each side of the core 173 to provide a 6:2:6 package substrate. Each accumulation layer 174 on one side of the substrate 170 may have a different function. For example, one or more of the layers 174 may be ground layers including conductive lines 177 connected to ground. One or more of the layers may be VDD layers including conductive lines 177 connected to VDD. One or more of the layers 174 may be signal layers that transmit signals between the connection components 160 and the fabric chip 140 and between the fabric chip 140 and the DRAM 110. In one example, two of the accumulation layers 174 are signal layers. The outermost layer 174 may include pads (not shown) for connection to other elements of the module 170.

[0126] 14a, vias 174c may be formed between accumulation layers 174 so that conductive lines 177 may pass between layers 174. Additionally, core vias 175 may be formed through core 173 so that conductive lines 177 may pass from the top side of substrate 170 to the bottom side.

[0127] Fabric chip 140 is flip chip affixed directly to substrate 170. In other words, fabric chip 140 is a semiconductor chip that is manufactured to include solder bumps on the face of the die. These bumps are then attached directly to substrate 170. In one example, the bump pitch in the core area of ​​the chip is approximately: x=261 microns y=154 microns Diagonal pitch = 151 microns where x is the width direction between the long edges of the chip and y is the length direction between the short edges of the chip. The pitch may be larger to provide connections to the DRAM 110, for example x=286 microns, y=164 microns, diagonal pitch=167 microns. The structure and function of the fabric chip 140 are described in more detail below with reference to FIG. 15.

[0128] The conductive lines 177 of the package substrate 170 are thin enough in the area of ​​the substrate 170 beneath each fabric chip 140 to allow the lines to break out of the chip 140 footprint.

[0129] The DRAMs 110 are mounted to the substrate 170 using a ball grid array (BGA). That is, the DRAMs 110 are each in the form of a packaged semiconductor chip including a die and a package substrate. The die is secured to the top side of the package substrate and is electrically connected to the package substrate. The package substrate has a grid of solder balls formed on the underside of the package substrate, which are then secured to corresponding conductive pads on the substrate 170. The BGA may have a pitch of, for example, 650 microns. Thus, the pitch of the balls is substantially coarser than the bumps of the fabric chip 140.

[0130] The connection components 160 may be connected to the substrate 170 via a BGA. Thus, each connection component 160 may include a grid of solder balls arranged on a face of the connection component 160 opposite the face including the pins 161.

[0131] Figure 15 illustrates in more detail the structure of fabric chip 140. Bumps formed on the underside of fabric chip 140 for connection to a substrate are illustrated as small circles generally designated 141. Blocks 142, 143, and 144 shown in Figure 15 illustrate areas within fabric chip 140 where specific circuitry may be placed.

[0132] The die of fabric chip 140 is rectangular and has two opposing long edges 140a, 140b and two opposing short edges 140c, 140d. The length of short edges 140c, 140d is approximately 6 mm. The length of long edges 140a, 140b is approximately 15 mm. In one example, each fabric chip is 5.5 mm by 15.3 mm. Thus, the aspect ratio of chip 140 is approximately 3:1. In one example, fabric chip 140 is a single or monolithic die.

[0133] Fabric chip 140 includes multiple memory controllers 142a-142h. Each memory controller 142 is a circuit formed on the die of the chip that serves as an interface to DRAM 100. In an example where the DRAM is an LPDDR DRAM, memory controller 142 is an LPDDR interface. The LPDDR interface meets the relevant LPDDR standard (e.g., JEDEC standard (e.g., JESD209-5B)). In an example where DRAM 110 is a different type of DRAM, memory controller 142 may meet the standard required to access DRAM 110 as appropriate.

[0134] As described above, each fabric chip 140 is associated with two DRAMs 110 on the top side and two DRAMs 110 on the bottom side 172. Each of these DRAMs 110 may have multiple memory channels (e.g., four memory channels). Thus, each DRAM 110 may be a four-channel DRAM. The fabric chip 140 includes a number of memory controllers 142 sufficient to access the channels of the DRAMs 110. In the illustrated example, the fabric chip 140 includes eight memory controllers 142a-142h, each of which is a dual-channel memory controller 142. This provides the necessary 16 channels to access four four-channel DRAMs. In one example, the memory channels are each 16 bits wide.

[0135] 15, the memory controller 142 is located on one long side 140a of the chip 140. Each chip 140 is placed on the module 100 so that side 140a faces the DRAM 100. This placement can reduce breakout of connections to the DRAM 100 from under the chip 140 footprint.

[0136] The memory controllers 142 may be arranged in a 2×4 grid with the first four of the memory controllers 142a-142d located closest to the long side 140a and the second four of the memory controllers 142e-142h located inside the first four memory controllers 142a-142d. The first four of the memory controllers 142a-142d may communicate with the DRAM 110 on the top side 171 of the substrate 110. The second four memory controllers 142e-142h may be arranged to access the DRAM 110 on the bottom side 172 through core vias 175 formed under the fabric chip 140. This arrangement also helps to mitigate breakout from underneath the footprint of the chip 140.

[0137] A number of bumps 141 of the fabric chip 140 are disposed under each memory controller 142 (at least some of which are memory attachment connectors) that are in electrical communication with the DRAM 110 via conductive lines 177 in the substrate 170. In other words, the bumps 141 under each memory controller include memory attachment ports connectable to the DRAM 110. Other bumps under each memory controller 142 may be connected to the memory controller's power supply or ground.

[0138] Each memory controller 142 forms part of the above-mentioned DDR interface block 48. Thus, the routing logic of the fabric chip 140 is configured to route data to and from the DRAM 110 through the memory controller 142.

[0139] The fabric chip 140 further includes a number of link controllers 143, 144. Each link controller 143, 144 may include circuitry formed on a die. A first group of link controllers 143 each include four communication lanes, labeled 143-1 through 143-4. Only one of the link controllers 143 is so labeled for clarity of the drawing.

[0140] Each communication lane 143-1-143-4 forms an individual communication link to an external device (i.e., a device not on module 100). Thus, at least some of the bumps 141 under each lane 143-1-143-4 of link controller 143 form an external link port or connector. The external link ports are connected to connection components 160 via conductive lines 177. The bumps 141 may include a bump for a transmit signal and a bump for a receive signal.

[0141] Each of the communication lanes 143-1 to 143-4 may be a serial link, such as a SERDES link as described above. Thus, the link controller 143 may include analog circuitry. In one example, the link controller 143 provides a 100 Gbps link.

[0142] In addition to the four communication lanes 143-1 to 143-4, link controller 143 may include a common area 143-5 that provides common functions to the four lanes 143-1 to 143-4 (eg, a common clock signal for each communication lane).

[0143] Three of the link controllers (labeled 143a-c) provide communication lanes for communication with each processor core 20. Thus, each fabric chip 140 has 12 communication lanes for communication with the processor core 20. As described above with respect to FIG. 9, three links (e.g., L2a, L2b, L2c) are provided to each processor core. Thus, each communication lane corresponds to an EPC (e.g., EPC2a, EPC2b, EPC2c) shown in FIG. 9. Thus, the 12 communication lanes on the fabric chip 140 provide connections to the four processor cores 20.

[0144] A further link controller (labelled 143d) provides four further communication lanes, which may include three lanes of pod-to-pod communication. The communication lanes may implement EPCs corresponding to Pa, Pb and Pc in FIG. 9 to implement pod-to-pod links PLa, PLb and PLc. The pod-to-pod links PL are sometimes referred to as cluster connecting links in that the pod-to-pod links connect fabric chips to different clusters.

[0145] Link controller 143d may include lanes for system-to-system communication. Thus, one of the communication lanes implements EPC (i.e., corresponds to PCS in FIG. 9) to provide a system link SL. The system link SL is connected to, for example, a switching fabric.

[0146] Link controller 144 provides a PCIe link to the host computer. Link controller 144 may include two sub-controllers (not shown) each implementing four lanes of communication. Link controller 144 may provide a lower speed connection compared to link controller 143.

[0147] The link controllers 143, 144 may be located along the long edge 140b opposite the memory controller 142. Again, this helps to mitigate breakout of the lines from underneath the fabric chip 140.

[0148] In one example, link controller 144 is only operable on one of the four fabric chips 140 on module 100. Link controllers 144 of the other three fabric chips 140 may not be connected to connection components 160 and therefore cannot communicate with the host.

[0149] The remaining bumps 141 of the fabric chip 140 that are not under one of the controllers 142, 143, 144 may include main chip power supply bumps and ground bumps. The main chip power supply may be a different power supply than the power supply of the memory controller 142. Additionally, some of the bumps 141 (e.g., bumps at the corners of the chip 140) may be dummy bumps that are not electrically connected. These dummy bumps are most susceptible to the different thermal expansion characteristics of the substrate 170 and the chip 140 and therefore cannot be used to reliably transmit signals.

[0150] FIG. 16 shows the two connection components 160a, 160b in more detail, with the other components of the module 100 omitted for clarity, apart from a box representing the position of the fabric chip 140 relative to the connection component 160.

[0151] Approximately half of the pairs of pins 161 are either VSS or ground pins. Thus, in each group of pins described below, approximately half of the pairs of pins perform the associated function and approximately half act as VSS.

[0152] The connection component 160 includes groups of pins 181 that carry signals to and from the link controller 143. These pins 181 may thus function as SERDES links. One group of pins 181 carries signals to and from a particular fabric chip 140. In particular, pins 181-1 carry signals to and from fabric chip 140-1, pins 181-2 carry signals to and from fabric chip 140-2, pins 181-3 carry signals to and from fabric chip 140-3, and pins 181-4 carry signals to and from fabric chip 140-4. The pins that form the SERDES links are generally located towards the outer edges of the connection component 160 (i.e., on the side of the connection component furthest from the DRAM 110).

[0153] Each group of pins 181 may include transmit pins 181a and receive pins 181b configured for transmitting and receiving signals, respectively. To maintain clarity of the drawing, only a selection of transmit pins 181a and receive pins 181b of pin 181-1 is labeled. The other groups of pins 181 include transmit pins 181a and receive pins 181b arranged similarly. In the example shown in FIG. 16, the transmit pins 181a and receive pins 181b are interspersed. In other words, the transmit pins 181a and receive pins 181b are distributed throughout the group 181, so that the transmit pin 181a may be located adjacent to (i.e., next to) the receive pin 181b. Adjacent or next to in this context includes pairs of pins arranged diagonally to each other.

[0154] Additionally, connection component 160 includes groups of pins 182 that carry power supplies for the fabric chips. In one example, the power supplies carried by pins 182 are the main digital power supplies for fabric chips 140. One group of pins 182 supplies a particular fabric chip 140 that has the group and the same subscript (i.e., -1, -2, etc.). Power supply pins 182 are typically located below the fabric chip 140 that they supply.

[0155] Connection component 160 includes a group of pins 183 in communication with link controller 144, which carries the PCIe link to the host computer. As mentioned above, only one of the fabric chips (e.g., fabric chip 140-1) may have a link controller 144 connected. Thus, only one of connection components 160a may include PCI link pins 183. Pins 183 are generally located toward the center of connection component 160a.

[0156] Connection component 160 further includes clock pins 184-1 through 184-4 that carry a clock signal for link controller 143 of each fabric chip 140. Connection component 160a may further include clock pin 185-1 for the PCIe controller. Only one clock pin 185-1 may be provided because only one fabric chip 140 has a PCIe link enabled.

[0157] Additionally, connection component 160 may include pin 186 that provides a power supply to link controller 143. In particular, pin 186 provides power to the PHY or analog components of link controller 143. Link controller 143 (in particular the PHY) may require a power supply that is different from the main digital power supply. For example, the link controller may require a power supply with less noise. Additionally, connection component 160 may include pin 187 that provides a power supply to DRAM 110.

[0158] 19 shows another example pair of connection components 260a, 260b. Connection component 260 operates similarly to connection component 160 with corresponding elements having corresponding reference numbers increased by 100. For clarity, only the differences will be detailed below.

[0159] The connection component 260 includes groups of pins 281 for transmitting signals to and from the link controller, corresponding in function to the pins 181 described above. In contrast to the pins 181, in each group of pins 281, the transmit pins 281a are separated from the receive pins 281b, i.e., the transmit pins 281a are not located adjacent to the receive pins 281b. In this example, adjacent includes diagonally adjacent pins.

[0160] For example, each group 281 may include an area including a transmit pin 281a and an area including a receive pin 281b. In other words, the transmit pin 281a and the receive pin 281b each form a continuous area. The two areas 281a and 281b are separated from each other by a group of ground pins 281c. The group of ground pins 281c forms a buffer or separation area between the transmit pin 281a and the receive pin 281b.

[0161] Furthermore, the transmit pins 281a associated with one fabric chip 240 are positioned so as not to be adjacent to the transmit pins 281a associated with another fabric chip 240. For example, the transmit pins 281a of group 281-1 associated with fabric chip 240-1 are positioned at or toward one end of the connection component 260, while the transmit pins 281a of group 281-2 associated with fabric chip 240-2 are positioned at or toward the opposite end of the connection component 260. However, the receive pins 281b of the two groups 281-1, 281-2 may be positioned adjacent to each other.

[0162] Advantageously, isolating the transmit pin 281a from the receive pin 281b has been found to reduce or prevent near-end crosstalk (NEXT) between the transmit pin 281a and the receive pin 281b. NEXT occurs due to the relative difference in signal strength between the transmit and receive signals, caused by signal degradation and attenuation as the signal travels between the transmit and receive points. Thus, the signal transmitted via the transmit pin 281a has a significantly higher gain than the signal received via the receive pin 281b, thus causing crosstalk.

[0163] Additionally, connection components 260a, 260b differ from connection components 160a, 160b in that they use pins located along the long edge of connection component 260 closest to DRAM 110 as DRAM power supply pins 287. It is understood that features of connection components 160 and 260 may be combined or interchanged.

[0164] Fig. 18 shows an example of a power supply arrangement for a module 100. As shown in Fig. 18, a motherboard 400 on which the module 100 can be mounted includes a power supply component 400P. The power supply component 400P includes, for example, load power supply points located on the opposite side of the module 100 and the motherboard 400. Power is supplied from the power supply component 400P to the fabric chips 140 via connection components 420 and 160.

[0165] Thus, the module 100 may not include a power supply (e.g., a point of load power supply). This allows the substrate 170 to be smaller, reducing the use of relatively high cost substrate material. Furthermore, positioning the fabric chip 140 on the opposite side of the substrate 170 directly above the connection components 160 minimizes the distance between the power supply components 400P and the fabric chip 140, reducing IR drop.

[0166] FIG. 17 illustrates a method of manufacturing an example module 100.

[0167] The method includes providing a substrate (S171). As described above, the substrate may be a package substrate 170 having a plurality of layers 174. The substrate may be formed by repeatedly arranging copper foil underlayers 174a and insulating underlayers 174b on a core 173. In some examples, layers 174 may be provided on both sides of the core 173. In some examples, vias 174c are formed between the layers and / or vias 175 are formed through the core 173, for example, by laser drilling.

[0168] In step S172, conductive lines 177 are formed on the substrate 170. In one example, the lines 177 are formed by etching the copper foil underlayer 174a. However, other methods may be used to form the conductive lines 177. The conductive lines 177 may pass through vias 174c, 175.

[0169] In step S173, a first semiconductor chip is attached directly to the substrate 170 by flip-chip attachment. The first chip may be a fabric chip 140. Solder bumps formed on the first chip may be attached to corresponding pads formed on the substrate 170 by heating the module 100. The module 100 may be heated by passing the module 100 through a suitable oven. Each of multiple first chips (e.g., four fabric chips 140) may be attached simultaneously, for example, in the same pass through an oven or other heating cycle.

[0170] In some examples, an underfill material (e.g., an epoxy resin) is underfilled onto the first chip. The underfill material may be delivered as a liquid by capillary action caused by a narrow gap formed between the underside of the chip and the substrate that draws the liquid between the solder bumps. The underfill may then be cured by a further heating cycle that is cooler than the heating cycle used to melt the solder bumps. The underfill helps to redistribute stresses caused by the different thermal expansion coefficients of the chip and the substrate.

[0171] In step S174, a packaged semiconductor chip having a BGA is attached to the substrate. The packaged semiconductor chip may be a DRAM 110 as described herein. The solder balls of the BGA of the packaged chip may be attached to corresponding pads formed on the substrate by, for example, heating the module 100 by passing the module 100 through a suitable oven. Heating the module 100 to attach the packaged semiconductor chip may be a separate heating cycle from the heating cycle that attaches the first semiconductor chip. Underfilling the first semiconductor chip may ensure that the first chip remains attached to the substrate during subsequent heating of the module to attach the packaged semiconductor chip.

[0172] Step S174 may include mounting a packaged semiconductor chip on one side of the substrate (e.g., top side 171) and then mounting a packaged semiconductor chip on the other side of the substrate (e.g., bottom side 172). Thus, there may be two separate heating cycles for mounting BGAs on each side of the substrate. In other examples, BGAs on both sides of substrate 170 may be mounted in a single heating cycle.

[0173] In step S175, the connection component 160 is attached to the substrate. In one example, the connection component 160 has a BGA. For example, the module 100 may be heated by passing the module 100 through a suitable oven to attach the solder balls of the BGA of the connection component to corresponding pads formed on the substrate. In some examples, the connection component 160 is attached to the substrate in an additional heating cycle following attachment of the packaged semiconductor chip. In other examples, the connection component 160 may be attached by the same heating cycle as one or more of the packaged semiconductor chips. For example, the connection component 160 may be attached to the underside 172 by the same heating cycle that attaches the packaged semiconductor chip to the substrate 170.

[0174] In some instances, the order of steps may be changed. The packaged semiconductor chip and / or the connecting components 160 may be secured to the substrate prior to flip-chip mounting the first chip.

[0175] Various modifications may be made to the module 100 described above. In some examples, the number of fabric chips 140 and DRAMs 110 present in the module 100 may differ from the examples described above. For example, each fabric chip 140 may be connected to fewer DRAMs 110 (i.e., one, two, or three DRAMs) or more DRAMs 110 (four or more DRAMs, e.g., eight DRAMs). In other examples, the module 100 may include fewer fabric chips 140 or more fabric chips 140. The module 100 may include fewer of the conceptual quadrants described above (e.g., two of the quadrants, six of the quadrants, eight of the quadrants, or any other suitable number). In some examples, the number of connection components 160 provided may vary. For example, only one connection component 160 or more than two connection components 160 may be provided. Additionally, the elements of the fabric chip 140 may vary. For example, fabric chip 140 may include more or fewer memory controllers and / or link controllers that communicate with a different number of DRAMs 110, processor cores 20, and other quads or pods. The locations of memory controllers 142 and link controllers 143, 144 may vary on fabric chip 140. The bumps of fabric chip 140 may have different functions and / or be arranged differently. The pins 161 of connection components 160 may be arranged differently and / or have different functions.

[0176] Advantageously, the module 100 provides routing capabilities and large capacity, high bandwidth and low latency memory for the processor cores 20, making it suitable for processing large machine learning models. When arranged as shown in FIG. 8 with a ratio of two modules 100 to each processor core 20, each processor can access up to 512 GB with an example bandwidth of 9.6 Tbit / s. Furthermore, the use of the module 100 limits the links required to the processor cores 20, conserving beachfront space that would normally be used for memory access and routing.

[0177] Advantageously, module 100 includes direct flip-chip attached fabric chip 140 and DRAM 110 and connection components 160 attached to module 100 via a BGA. By directly flip-chip attaching fabric chip 140 to the substrate, fabric chip 140 does not require additional packaging, thus allowing the overall size of module 100 to be reduced.

Claims

1. a memory and routing module, A substrate; a connection component mounted on the substrate, the connection component including a plurality of pins for mating the memory and routing module with a corresponding connection component on a motherboard; a dynamic random access memory (DRAM) chip disposed on the substrate; a routing chip mounted on the substrate, A memory controller; a plurality of connections including a first group of connections between the memory controller and the DRAM chips and a second group of connections with the pins of the connection component; routing logic configured to route data between the second group of connections and the first group of connections; a routing chip including: A memory and routing module including:

2. 2. The memory and routing module of claim 1, wherein the DRAM chip forms part of a DRAM chip package that includes a ball grid array (BGA) that connects the DRAM chip package to the substrate.

3. 3. The memory and routing module of claim 1 or 2, wherein the routing chip includes a plurality of bumps that secure the routing chip directly to the substrate by flip-chip attachment.

4. 4. The memory and routing module of claim 1, further comprising a plurality of DRAM chips disposed on the substrate, the routing chip including a plurality of memory controllers, each memory controller connected to a respective DRAM chip of the plurality of DRAM chips via the plurality of connections.

5. 5. The memory and routing module of claim 4, further comprising four DRAM chips connected to said routing chip.

6. 6. The memory and routing module of claim 4 or 5, wherein at least one of the plurality of DRAM chips is disposed on a first side of the substrate and at least one of the plurality of DRAM chips is disposed on a second side of the substrate.

7. 7. The memory and routing module of claim 1, further comprising a plurality of routing chips and a plurality of DRAM chips disposed on the substrate, each routing chip including a memory controller connected to each DRAM chip.

8. The memory and routing module of any one of claims 1 to 7, wherein the connection component is a mezzanine connection component.

9. the routing chip is disposed on a first side of the substrate; the connection components are mounted on a second side of the substrate at locations corresponding to locations of the routing chips; 9. The memory and routing module of claim 1, wherein the routing chip is configured to receive power via the connection component from a power supply component electrically coupled to the connection component.

10. The memory and routing module of claim 9 including a plurality of vias in said substrate forming electrical paths connecting said connection components to said routing chips.

11. The memory and routing module of any one of claims 1 to 10, wherein the DRAM chips are low power double data rate (LPDDR) memory and the memory controller includes a LPDDR interface.

12. The memory and routing module of any one of claims 1 to 11, wherein the memory controller is located on a long edge of a die of the routing chip, the long edge of the die facing the DRAM chip.

13. the second group of connections includes a plurality of processor connections each configured to attach to a respective processor chip; the routing logic is configured to route a signal from one of the processor connections to another of the processor connections; A memory and routing module according to any preceding claim.

14. 14. The memory and routing module of claim 13, wherein the processor connection is a serializer / deserializer (SERDES) connection.

15. the plurality of pins includes processor connection pins configured to support the plurality of processor connections; the processor connection pins include a transmit pin and a receive pin; The receiving pins are arranged so as not to be adjacent to the transmitting pins.

15. A memory and routing module according to claim 13 or 14.

16. 16. The memory and routing module of claim 1, wherein the substrate is a package substrate including a plurality of conductive lines, and the first group of connections and the second group of connections are via the plurality of conductive lines.

17. A system comprising a memory and routing module according to any one of claims 1 to 16, and a number of processor chips connected to the memory and routing module via connection components.

18. Each processor chip does not communicate directly electrically with the other processor chips, Each processor chip is configured to communicate with the other processor chips via a routing chip; 20. The system of claim 17.

19. 19. The system of claim 17 or 18, wherein the multiple processor chips perform time-deterministic processing.

Images