Multi - column interleaved DIMM layout and routing topology

Abstract

In one embodiment, a printed circuit board (PCB) has a first central processing unit (CPU) socket and a second CPU socket that is substantially aligned with the first CPU socket, and also has a first plurality of dual in-line memory module (DIMM) sockets interconnected with the first CPU socket and a second plurality of DIMM sockets (in a direction parallel to the first plurality of DIMM sockets) interconnected with the second CPU socket. The first plurality of DIMM sockets are arranged in at least a first column of DIMM sockets and a second column of DIMM sockets on the PCB, and the second plurality of DIMM sockets are arranged in at least the second column of DIMM sockets and a third column of DIMM sockets on the PCB such that the second column of DIMM sockets contains interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 398,861, titled MULTI-COLUMN INTERLEAVED DIMM PLACEMENT AND ROUTING TOPOLOGY, filed Apr. 30, 2019, by Hoi San Leung, the content of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to computer architectures, and more particularly, to multi-column interleaved dual in-line memory module (DIMM) placement and routing topologies. Background Art

[0004] Data centers use servers to provide the computing resources (e.g., processing, memory space, network, and disk I / O, etc.) required for workload operation. With the explosion of workloads and the increase in computing demands, it is necessary to expand or "scale out" server resources to meet the growing demands. There are two ways to scale out server resources in a data center. The first is to add more servers or "scale out horizontally". For example, assume an enterprise has a virtual server running five applications and using 80% of the computing power of a physical server. If the enterprise needs to deploy more workloads and the physical server lacks sufficient computing power to support the additional workloads, the enterprise may need to deploy additional servers to support the new workloads. A scale-out architecture is also known as a cluster or a perturbed computing method, where multiple small servers share the computing load of a single application. For example, a mission-critical workload may be deployed on two or more servers, and the processing is shared among these servers so that if one server fails, another can take over and maintain the availability of the application. If more redundancy is needed, additional server nodes can be used to scale out the cluster.

[0005] Advances in technology as well as server computing power have increased the amount of resources that a single server can provide. Today's servers have much more processing, memory, and I / O capabilities than previous models in a similarly sized chassis. This approach is known as "scaling up", because a physical server can handle more and / or larger workloads. Referring again to the above example, using the scale-up method, a new server with much more computing resources can be deployed in the next technology update cycle, migrating all workloads from the old server to the new server, taking the old server out of service or assigning it to other tasks, and leaving significantly more available resources to handle additional production workloads without significantly increasing the data center space or energy requirements.

[0006] As a specific example, server memory requirements have been increasing, and CPU vendors are adding more and more memory channels to meet this demand. However, unless there is space (or "real estate") on the printed circuit board (PCB) to accommodate additional memory slots, such as for dual in-line memory modules (DIMMs), the additional memory channels may not be usable. This is especially more difficult in the case of using a smaller PCB, such as for a half-width form factor processor blade. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments herein can be better understood with reference to the following description in conjunction with the accompanying drawings, in which like reference numerals represent like or functionally similar elements, wherein:

[0008] FIG. 1 is a simplified block diagram showing an example symmetric multi-processing ("SMP") system that may be deployed in embodiments described herein;

[0009] Figure 2 is a simplified block diagram showing a conventional arrangement of a 2-way SMP system including two processors disposed on a wide motherboard;

[0010] Figure 3 is a simplified block diagram showing an arrangement of a 4-way SMP system according to features of embodiments described herein for implementing an interconnection method for a scale-out server;

[0011] Figure 4 is a simplified block diagram showing a processor complex and a connection to a platform controller hub (PCH) that may be deployed in embodiments described herein;

[0012] Figure 5 is a simplified block diagram showing the difficulty of adding a memory socket to a conventional processor complex topology;

[0013] Figure 6 is a simplified block diagram showing a multi-column interleaved dual in-line memory module (DIMM) layout and routing topology according to one or more embodiments described herein;

[0014] Figure 7 is a simplified block diagram showing an example of a portion of a stack of printed circuit boards (PCBs) comparing a conventional PCB stack with a PCB stack for a multi-column interleaved DIMM topology according to one or more embodiments described herein;

[0015] Figures 8A - 8C is a simplified block diagram showing an example layout diagram of portions of a PCB stack for a multi-column interleaved DIMM topology according to one or more embodiments described herein; and

[0016] Figure 9is a simplified block diagram showing components of an example machine capable of executing instructions in a processor of a system to implement various features of the embodiments described herein. Detailed Description

[0017] Overview

[0018] Aspects of the invention are set out in the independent claims and the preferred features are set out in the dependent claims. Features of one aspect may be applied to each aspect alone or in combination with other aspects.

[0019] According to one or more embodiments of the present disclosure, a printed circuit board (PCB) has a first central processing unit (CPU) socket and a second CPU socket that is substantially aligned with the first CPU socket. The PCB also has a first plurality of dual in-line memory module (DIMM) sockets interconnected with the first CPU socket and a second plurality of DIMM sockets interconnected with the second CPU socket, the first plurality of DIMM sockets being located on opposite first and second sides of the first CPU socket, and the second plurality of DIMM sockets being located on opposite first and second sides of the second CPU socket and in a direction parallel to the first plurality of DIMM sockets. The first plurality of DIMM sockets are arranged in at least a first column of DIMM sockets and a second column of DIMM sockets on the PCB across both opposite first and second sides of the first CPU socket. The second plurality of DIMM sockets are arranged in at least a second column of DIMM sockets and a third column of DIMM sockets on the PCB across both opposite first and second sides of the second CPU socket such that the second column of DIMM sockets contains interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets.

[0020] According to one or more additional embodiments of the present disclosure, the first column and the second column are separated by a center line of the first CPU socket, and the second column and the third column are separated by a center line of the second CPU socket. In one embodiment, a portion of the first plurality of DIMM sockets closest to the first CPU socket is further disposed on the center line of the first CPU socket on the PCB, and a portion of the second plurality of DIMM sockets closest to the second CPU is further disposed on the center line of the second CPU socket on the PCB.

[0021] Other embodiments are described below, and this summary is not intended to limit the scope of the present disclosure.

[0022] Description

[0023] As described above, there are two main types of server implementations used in data centers; namely, scale-up and scale-out. Both types of servers use multiple processors. In a scale-up server, the processors are interconnected via cache coherence links and all work together under a single operating system software. This way of connecting multiple processors together can also be referred to as symmetric multiprocessing (“SMP”), and the cache coherence links can be called SMP links. Scale-up servers are typically (but not always) used for applications in high-performance databases, analytics, and compute servers. On the other hand, scale-out servers do not use cache coherence links between processors, and each processor subsystem works as an independent server with its own operating system software.

[0024] The implementation of scale-up servers is particularly challenging. The cache coherence links between any two processors require very high-bandwidth (data rate) interconnections. These interconnections are implemented as bundles by multiple parallel high-speed serializer / deserializer (“SERDES”) channels, which are called “ports”. SERDES is a pair of functional blocks commonly used for high-speed communication to compensate for limited I / O. The SERDES blocks convert data between serial and parallel interfaces in each direction. The main purpose of SERDES is to provide data transmission through a single / differential line to minimize the number of I / O pins and interconnections. The SERDES function includes two functional blocks, including a parallel input serial output (“PISO”) block (also called a parallel-to-serial converter) and a serial input parallel output (“SIPO”) block (also called a serial-to-parallel converter). The PISO block can include a parallel clock input, a set of data input lines, and input data latches. An internal or external phase-locked loop (“PLL”) can be used to multiply the input parallel clock by the serial frequency. The simplest form of PISO has a single shift register that receives parallel data once for each parallel clock and shifts it out at a higher serial clock rate. The SIPO block can include a received clock output, a set of data output lines, and output data latches. A serial clock recovery technique can be used to recover the received clock from the data. SERDES that do not transmit a clock use a reference clock to lock the PLL to the correct TX frequency, avoiding low harmonic frequencies that may occur in the data stream. The SIP block divides the input clock to the parallel rate. A typical implementation has two registers connected as a double buffer. In such an implementation, one register is used to time the serial stream and the other is used to hold the data at the slower parallel side.

[0025] A typical implementation may employ 20 or more SERDES channels per port. For bidirectional communication, there may be a transmit port (“TX port”) and a receive port (“RX port”). Since a scale-out system includes multiple processors, each processor will have multiple TX / RX port pairs, and each port will in turn have multiple high-speed SERDES channels. This poses a significant interconnection challenge in the system. The embodiments described herein address this interconnection challenge in designing a modular scale-out server system. In particular, the embodiments described herein use a 2-way symmetric multi-processor (“SMP”) system implementation as a building block to implement 4-way and 8-way (and more) SMP systems. An SMP system is a multi-processor system that has a centralized shared memory, referred to as the “main memory,” and operates under a single operating system with two or more homogeneous processors.

[0026] Figure 1 FIG. shows a simplified block diagram of an exemplary conventional SMP system 10. As Figure 1 shown, an SMP system (such as SMP system 10) is a tightly coupled multi-processor system that has a pool of homogeneous processors 12 that operate independently, each processor executing a different program on different data, having the ability to share resources (such as main memory 14 and I / O 16), and being connected via a system bus or crossbar 18. Each processor 12 is typically associated with a dedicated cache memory 20 to accelerate main memory data access and reduce system bus traffic. A bus arbiter 22 is provided to arbitrate access by the processors 12 to the system bus 18.

[0027] According to the features of the embodiments described herein, a 2-way SMP system implementation uses only half the width of a conventional 2-way system used to implement 4-way and 8-way systems. In some embodiments, this half-width 2-way SMP system is implemented by placing the processors one after another along the length of a printed circuit board (referred to as shadow core placement). As Figure 2 shown, a conventional or traditional 2-way SMP system 30 includes two processors or CPUs 32 that are arranged on a wide motherboard 34. Each CPU 32 includes multiple cache-coherent SMP links 36, all of which are connected to corresponding connectors 38 arranged on a midplane or backplane of the server system 30 that is Figure 2 collectively designated by reference numeral 40 in the drawings. As will be described in more detail below, Figure 2 the method shown in the drawings requires a motherboard that is twice as wide as the half-width SMP implementation described herein. System 30 is not configurable for 4-way and 8-way SMP implementations; thus, 2-way, 4-way, and 8-way SMP implementations require separate motherboard designs.

[0028] In certain embodiments described herein, cache coherence links are split such that half of them go into the midplane (or backplane), while the other half go into the front panel disposed on the front side of the system chassis. This approach only requires smaller-sized connectors on the front and back sides, thereby enabling good air flow and regulating the operating temperature of the processors and their subsystems. This arrangement also significantly reduces the wiring density requirements, enabling the SMP links to be routed in fewer printed circuit board layers, as opposed to implementations that bring all cache coherence links to the front or back side. This arrangement is shown in Figure 3 which shows a 4-way SMP system 50 according to the features of the embodiments described herein, for implementing an interconnection method for scale-out servers.

[0029] As Figure 3 shown, the SMP system 50 includes four CPUs 52(1)-52(4), where two (i.e., CPU 52(1) and 52(2)) are disposed on a first motherboard 54(1), and two (i.e., 52(3) and 52(4)) are disposed on a second identical motherboard 54(2). Each of the CPUs 52(1)-52(4) has a plurality of cache coherence links that interconnect each CPU to another CPU. According to the features of the embodiments described herein, half of the cache coherence links in the system (i.e., the links connecting CPU 52(1) and 52(4) and the links connecting CPU 52(2) and 52(3)) are connected to protocol-independent electrical switching drivers 60, 61 on the front panel 62 of the system 50 chassis, and the other half of the cache coherence links (e.g., the cache coherence links connecting CPU 52(2) and CPU 52(4) and the cache coherence links connecting CPU 52(1) and 52(3)) are connected to switching drivers 64, 65 on the backplane (or midplane) 66 of the system 50 chassis. In some implementations, protocol-independent electrical switching drivers may not be required, in which case the cache coherence links will be directly connected to the connectors. As a result, and according to the features of the embodiments described herein, the width of each of the motherboards 54(1) and 54(2) is half the width of the motherboard 34( Figure 2 ). It will be appreciated that each of the motherboards 54(1), 54(2) may correspond to a blade server or a "blade".

[0030] The cache coherence links connected to the front panel and the back panel are carefully selected so that the system can be easily reconfigured for 4-way and 8-way SMP implementations, with a maximum distance of one hop between processors. The cache coherence links connected to the front and back sides tend to be longer, thus resulting in high signal loss and causing signal integrity issues. To avoid signal integrity issues, protocol-independent broadband electrical signal amplifiers or retiming drivers (such as retiming drivers 60, 61, 64, 65) can be deployed to compensate for the loss caused by the long length of the interconnect medium. Alternatively, some embodiments can utilize twinaxial cables ("twinax") or coaxial cables to connect the cache coherence links to the back panel and the front / midplane without using retiming drivers (only the PHY layer). Such twinax or coaxial cables can also be selectively used for the TX / RX ports with the highest losses, while other TX / RX ports are routed on the printed circuit board or the motherboard. The cables can be attached to the printed circuit board through high-density connectors or directly crimped to the through-holes of the printed circuit board. The connectors and / or through-holes can be oriented parallel to the airflow direction to avoid blocking it. (Note that a "through-hole" is a copper tube used to connect signals from different wiring layers or to connect the static power supply and ground to the respective planes on the PCB.)

[0031] Generally, the high-speed links connected to the backplane / midplane and / or the front panel carry cache coherence SMP data. However, in some scenarios where a multi-way SMP system is not required, the same high-speed links can be used to carry processor I / O traffic (such as PCIe data) or network data to expand the system capabilities such as storage. Implementing 4-way and 8-way (and above) SMP systems in smaller form factors for achieving higher server density is particularly challenging. The embodiments described herein implement a method for interconnecting SMP links in a modular 2-way processor subsystem, which is in turn used to construct 4-way and 8-way systems implemented in a compact form factor.

[0032] The embodiments described herein can achieve a higher server density than traditional implementations; in particular, in a 6U form factor, up to four 4-way systems and up to two 8-way systems, while traditional systems require a 10U or higher form factor to achieve a comparable number of comparable systems. In addition, the system can be easily reconfigured for 2-way, 4-way, and 8-way SMP implementations. It requires fewer printed circuit board layers, thus reducing the cost of the server motherboard, and compared to implementations based on node concentrators or multiplexers up to 8S, the subject embodiments can achieve lower latency and higher performance.

[0033] Figure 4A simplified example of a printed circuit board (PCB) (e.g., a motherboard) 54 is shown, in which a processor complex with two processor sockets for CPU-1 52(1) and CPU-2 52(2) is interconnected with an interconnect link 67, where one processor (e.g., CPU-1) is directly connected to a platform controller hub (PCH) 69 via a link 68, and a second processor (e.g., CPU-2) accesses the PCH via an interface link 67 between the two CPUs. As shown, each processor has a set of DIMMs 70, listed in pairs for each processor as 1-A1 / A2, 1-B1 / B2, 1-C1 / C2, and 1-D1 / D2 for CPU-1, and 2-A1 / A2, 2-B1 / B2, 2-C1 / C2, and 2-D1 / D2 for CPU-2. Thus, Figure 4 the half-width PCB 54 in Figure 4 can install a total of 24 DIMMs, i.e., 12 (twelve) DIMMs per CPU (i.e., 6 (six) "memory channels" are used per socket - where a memory channel is a set of independent interfaces connecting the CPU to the DIMMs, and each memory channel holds two (or three) DIMMs). Note that the DIMMs may be RDIMMs, LRDIMMs, 3DS LRDIMMs, etc., with DDR4 or DDR5 or higher speeds (as developed).

[0034] —Multi - column Interleaved DIMM Layout and Routing Topology—

[0035] Current PCB technology uses several different types of topological layouts to place DIMM connectors (e.g., daisy chain, tee, star, etc.) to maximize the number of DIMM connectors per memory channel to achieve high memory capacity while operating at the highest possible frequency that the memory channel can support. However, as more and more DIMM connectors are added to the PCB to obtain higher memory capacity, it becomes increasingly difficult to keep the trace lengths as short as possible because the signal traces connecting the additional DIMM connectors are lengthened in traditional topologies. Additionally, the DIMM connector farthest from the multi-core socket typically sets the speed at which the memory bus can operate, so adding more memory slots traditionally means slower memory speed.

[0036] In particular, double data rate (DDR) synchronous dynamic random access memory (SDRAM) has been continuously evolving over the years to achieve increasingly higher speeds across generations (DDR, DDR2, DDR3, DDR4, DDR5, and more generations may still be in development). DDR4 / 5 designs operating at 2933 MT / s or higher have very strict design requirements, such as: maximum trace length, trace length matching within a byte group, trace-to-trace spacing, trace-to-VIA spacing, DIMM-to-DIMM spacing, PCB thickness, and GND / PWR references for Data and Address / Cmd / Ctrl lines. These design issues become more complex due to additional memory channel and DIMM requirements. For example, the data bits DQ[7:0] have 8 nets that need to be length-matched with the strobe nets (DQS, typically two differential pairs per byte group in high-end CPUs). If the DIMMs are not placed correctly, it will significantly increase the trace length and the difference in trace length between the nets, requiring more PCB real estate for length matching (or more routing layers), ultimately increasing the board thickness (mechanical design constraint). In addition, longer channel lengths increase insertion loss and crosstalk (both near-end and far-end), thus reducing the timing and voltage margins. Finally, if the DIMMs are not placed properly, it will be extremely challenging to provide ground (GND) and power (PWR) references on the Data and CLK (clock) / Cmd / Add (address) / Ctrl groups, thereby increasing the design risk.

[0037] In addition, as mentioned above, server memory requirements have been increasing, and CPU vendors are adding more and more memory channels to meet this demand. However, the space limitations of printed circuit boards (PCBs), especially for smaller PCBs (e.g., half-width), make it difficult to accommodate additional dual in-line memory modules (DIMMs). The traditional single-row DIMM placement method (as shown above) driven by CPU pins places further restrictions on the design, making it impossible to utilize the additional memory channel capacity. For example, Figure 4 shown Figure 5 how the single-row topology design Figure 4 cannot accommodate additional DIMMs on either side of the CPU (i.e., without increasing the size of the PCB, which is fixed in many design applications, the additional DIMMs will exceed the board boundary of the PCB54).

[0038] Accordingly, the present disclosure provides a novel multi-column interleaved DIMM layout and routing topology by combining two unique design aspects described in more detail below, namely, a multi-column DIMM layout and DIMM interleaving (from different CPU groups). In particular, the multi-column interleaved DIMM topology here overcomes the limitations imposed by traditional single-column DIMM topologies due to PCB real estate constraints and fully utilizes the increased memory bandwidth and CPU capabilities while still meeting the necessary design constraints. For example, an illustrative result of the embodiments described below is a half-width PCB that can accommodate 16 (sixteen) DIMMs per CPU (eight memory channels per socket), for a total of 32 DIMMs, without increasing the size of the half-width blade. (Other sizes and total DIMM counts can also be achieved using the techniques here, and those mentioned here are just illustrative examples.)

[0039] In operation, Figure 6 An example configuration for a multi-column interleaved DIMM is shown in accordance with features of embodiments herein, where the white DIMMs 81 belong to CPU-1 52(1) and the shaded DIMMs 82 belong to CPU-2 52(2). This unique design feature facilitates the installation of all 32 DIMMs despite the very limited board space (e.g., half-width board) of the embodiment.

[0040] In particular, as Figure 6 shown, a printed circuit board (PCB) 54 has a first central processing unit (CPU) socket 52(1) on the PCB and a second CPU socket 52(2) that is substantially aligned with the first CPU socket. In addition, the PCB has a first plurality of dual in-line memory modules (DIMMs) 81 that are interconnected with the first CPU socket 52(1) (e.g., via memory channels), where the first plurality of DIMM sockets are located on opposite first and second sides of the first CPU socket, as shown. A second plurality of DIMM sockets 82 on the PCB can be interconnected with the second CPU socket 52(2), where the second plurality of DIMM sockets are also located on opposite first and second sides of the second CPU socket and in a direction parallel to the first plurality of DIMM sockets. Note that the first plurality of DIMM sockets and the second plurality of DIMM sockets are configured for one of a plurality of double data rate (DDR) fourth generation (DDR4) DIMM sockets or DDR fifth generation (DDR5) DIMMs or any other suitable DIMM configuration.

[0041] According to an embodiment of the present disclosure, a first plurality of DIMM sockets 81 (spanning opposite first and second sides of the first CPU socket) are arranged in DIMM sockets of at least a first column 83 and a second column 84 on the PCB, while a second plurality of DIMM sockets 82 (spanning opposite first and second sides of the second CPU socket) are arranged in DIMM sockets of at least the second column 84 and a third column 85 on the PCB, such that the DIMM sockets in the second column include interleaved DIMM sockets from each of the first plurality of DIMM sockets 81 and the second plurality of DIMM sockets 82.

[0042] In an example embodiment, the first column 83 and the second column 84 are separated by a center line 86 of the first CPU socket 52(1), and the second column 84 and the third column 85 are separated by a center line 87 of the second CPU socket 52(2). In a specific embodiment, a portion (81-a) of the first plurality of DIMM sockets closest to the first CPU socket is further disposed on the center line of the first CPU socket on the PCB, and a portion (82-a) of the second plurality of DIMM sockets closest to the second CPU socket is further disposed on the center line of the second CPU socket on the PCB.

[0043] Using the above illustrative paired naming convention, a portion 81-a of the first plurality of DIMM sockets closest to the first CPU socket includes two DIMM sockets located on the first side of the first CPU socket, namely 1-C1 / C2, and two DIMM sockets located on the second side of the first CPU socket, namely 1-G1 / G2. Conversely, a portion 82-a of the second plurality of DIMM sockets closest to the second CPU socket includes two DIMM sockets located on the first side of the second CPU socket, namely 2-G1 / G2, and two DIMM sockets located on the second side of the second CPU socket, namely 2-C1 / C2. (Note that the pins of the first CPU socket and the second CPU socket may be exemplarily rotated 180 degrees relative to each other, so the "first side" of the CPU socket and the "second side" of the CPU socket correspond to the same side of the PCB, but electrically, the pins on the first side of the first CPU socket will correspond to the pins on the second side of the second CPU socket.)

[0044] The first plurality of DIMM sockets 81 in the first column 83 may illustratively include four DIMM sockets located on the first side of the first CPU socket 52(1), such as 1-B1 / B2 and 1-D1 / D2, as shown, and four DIMM sockets located on the second side of the first CPU socket, such as 1-F1 / F2 and 1-H1 / H2. Similarly, the second plurality of DIMM sockets 82 in the third column 85 includes four DIMM sockets located on the first side of the second CPU socket 52(2), such as 2-F1 / F2 and 2-H1 / H2, and four DIMM sockets located on the second side of the second CPU socket, such as 2-B1 / B2 and 2-D1 / D2. In this example, the first plurality of DIMM sockets 81 in the second column 84 includes two DIMM sockets located on the first side of the first CPU socket 52(1), such as 1-A1 / A2, and two DIMM sockets located on the second side of the first CPU socket, such as 1-E1 / E2. Additionally, the second plurality of DIMM sockets 82 in the second column 84 includes two DIMM sockets located on the first side of the second CPU socket 52(2), such as 2-E1 / E2, and two DIMM sockets located on the second side of the second CPU socket, such as 2-A1 / A2.

[0045] Based on the pin design of the CPU socket (the way pins on the CPU are assigned or grouped for connection), the memory interface pins of a particular channel typically cluster together for connection purposes. Thus, the illustrative layout of the DIMMs as Figure 6 shown provides maximum performance while having a minimal impact on functional constraints. For example, the memory channels interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets may dictate that the pins of the first CPU socket and the second CPU socket be rotated to a direction that provides the shortest length for the longest memory channel in the memory channels (out of the four possible rotation directions, i.e., 90-degree increments). Additionally, the memory channels interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets are configured based on an illustrative design such that the longest memory channel (e.g., to the DIMM at the PCB edge) is less than or equal to the required 6.8 inches.

[0046] It is worth noting that the PCB layer stack design also plays an important role in system design, especially for high-performance CPUs where the memory speed is typically highest. Certain design constraints must be met when designing the stack, as shown in Table 1 below. These constraints are driven by mechanical layout and topology, thermal airflow, and electrical parameters such as propagation delay, impedance, and crosstalk.

[0047] <![CDATA Constraints > <![CDATA Parameters > PCB Board Thickness 140 mils (max) Copper Foil Thickness 1 oz for signals; 2 oz for power PCB Dielectric Constant (DK) ～3.4-4.0 Dissipation Factor (Df) ～0.0013 Power Plane Clearance Shared by Clock Groups Channel Length 2.5 - 7.0” (max) Routing Layers 22 (max) DIMM Spacing 340 mils

[0048] Table 1

[0049] Figure 7 An example showing a portion of a stack of PCBs (e.g., showing 8 out of an illustrative 22 layers) compares a conventional PCB stack (left, 105) with an example (non-limiting) PCB stack (right, 110) for a multi-column interleaved DIMM topology used herein. It can be seen that stack 110 is different from conventional stack 105. Notably, the near references of G02 and G06 are used to minimize the plane voids required for the CLK group signals (address, cmd, Ctrl), thereby minimizing the number of layers and thus minimizing the board thickness (the conventional stack 105 requires multiple plane voids for clock group routing, as shown by G02, G04, and G06). Additionally, this design helps keep the PCB thickness less than the required 140 mils.

[0050] Specifically, the near references regarding G02 and G06 are used to minimize the plane voids required for the CLK group signals, thereby minimizing the number of layers and thus minimizing the board thickness, as Figures 8A - 8C shown, for example, a simplified layout of the respective layers of an illustrative PCB is shown according to the techniques herein, where in Figure 8A , layer G02 (122) of the entire upper PCB 54 is shown to have a plane void 132 for VDDQ (power voltage to the output buffers of the memory chips, i.e., to pins 120 of DIMMs 81 / 82) to the innermost DIMMs 81-a and 82-a. (Note that most of the remaining space on the layer (not dedicated to other circuits) may be ground (GND).) As Figure 8B shown, due to the illustrative stack design here, layer G04 (124) is shown to have no plane void for VDDQ. Finally, in Figure 8C , layer G06 (126) is shown to have a plane void 136 for VDDQ to the corresponding DIMM pin 120. On the other hand, a conventional layer stack would fill the GND plane voids with VDDQ (power) for layers G02, G04, and G06. (Note that for brevity, other layers are not shown, but further layout efficiencies can be established based on the illustrative layout described herein.)

[0051] Thus, the above design achieves the goal of placing more DIMMs on the PCB to utilize all memory channels while meeting the electrical, mechanical, and thermal constraints in the design. That is, the multi-column staggered DIMM design optimizes the use of PCB space to achieve a full 32-DIMM configuration (for maximum performance) while maintaining support for all I / O devices on the illustrative half-width board. This design specifically considers avoiding any mechanical constraint violations (e.g., component placement / topology, board thickness, DIMM spacing, CPU and DIMM orientation, etc.), while also meeting the thermal condition requirements and minimizing (e.g., avoiding) any power distribution impacts. In addition, the multi-column staggered DIMM design also meets electrical limitations, such as channel routing length, stub length between DIMMs (or DIMM spacing) (where "stub" is additional electrical delay that degrades signal quality on trace stubs or VIA stubs), via length (related to board thickness), proper plane reference on Strobe and CLK groups, crosstalk minimization, signal quality meeting voltage and timing margins, etc.

[0052] Thus, the techniques described herein provide a multi-column staggered DIMM layout and routing topology. In particular, the embodiments herein help to expand the DIMM topology on a PCB with more memory channels (increasing bandwidth), where the multi-column design together with the staggered DIMM addresses the numerous design challenges detailed above, significantly reducing the design risk.

[0053] It should be noted that it will be recognized that the various blades shown in the figures can be implemented using one or more computer devices that include software embodied in one or more tangible media for facilitating the activities described herein. The computer devices used to implement the blades can also include storage devices (or storage elements) for storing information to be used when implementing the functions outlined herein. In addition, the computer devices used to implement the blades can include one or more processors capable of executing software or algorithms to perform the functions discussed in this specification. These devices can also, where appropriate and based on specific requirements, save information in any suitable storage element (random access memory ("RAM"), ROM, EPROM, EEPROM, ASIC, etc.), software, hardware, or any other suitable component, device, element, or object. Any memory item discussed herein should be construed as being included within the broad term "storage element". Similarly, any potential processing element, module, and machine described in this specification should be construed as being included within the broad term "processor". Each network element can also include a suitable interface for receiving, sending, and / or otherwise conveying data or information in a network environment.

[0054] Note that, in some example implementations, the various functions outlined herein can be implemented by logic encoded in one or more tangible media (e.g., embedded logic provided in a special - purpose integrated circuit (“ASIC”), digital signal processor (“DSP”) instructions, software (which may include object code and source code) executed by a processor or other similar machine, etc.). In some cases of these instances, the storage element can store data for the operations described herein. This includes storage elements capable of storing software, logic, code, or processor instructions that are executed to perform the activities described in this specification. The processor can execute any type of instructions associated with the data to implement the operations detailed in this specification. In one example, the processor can transform an element or article (e.g., data) from one state or thing to another. In another example, the activities outlined herein can be implemented with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor), and the elements identified herein can be a type of programmable processor, programmable digital logic (e.g., field - programmable gate array (“FPGA”), erasable programmable read - only memory (“EPROM”), electrically erasable programmable ROM (“EEPROM”)), or an ASIC that includes digital logic, software, code, electronic instructions, or any suitable combination thereof.

[0055] It should be noted that most of the infrastructure discussed herein can be provided as part of any type of network element. As used herein, the term “network element” or “network device” can include a computer, server, network device, host, router, switch, gateway, bridge, virtual device, load balancer, firewall, processor, module, or any other suitable device, component, element, or object operable to exchange information in a network environment. Additionally, a network element can include any suitable hardware, software, component, module, interface, or object that facilitates its operation. This may include appropriate algorithms and communication protocols that allow for the efficient exchange of data or information.

[0056] In one implementation, a network element / device can include software for implementing (or facilitating) the management activities discussed herein. This can include the implementation of instances of any of the components, engines, logic, etc. shown in the figures. Additionally, each of these devices can have an internal structure (e.g., a processor, storage element, etc.) to facilitate some of the operations described herein. In other embodiments, these management activities can be performed external to these devices, or included in some other network element to achieve the desired functionality. Alternatively, these network devices can include software (or interactive software) that can coordinate with other network elements to implement the management activities described herein. In other embodiments, one or more devices can include any suitable algorithms, hardware, software, component, module, interface, or object that facilitates its operation.

[0057] Go to Figure 9 , which shows a simplified block diagram of an example machine (or apparatus) 210, which in some embodiments may include one of the blade servers shown herein and which may be implemented in the embodiments shown and described with reference to the accompanying drawings provided herein. The example machine 210 corresponds to network elements and computing devices that may be deployed in the environments described herein. In particular, Figure 9 shows a block diagram representation of an example form of a machine, where software and hardware cause the machine 210 to perform any one or more of the activities or operations discussed herein. As Figure 9 shown, the machine 210 may include a processor 212, a main memory 213, an auxiliary memory 214, a wireless network interface 215, a wired network interface 216A, a virtual network interface 216B, a user interface 217, and a removable media drive 218 (including a computer-readable medium 219). A bus 211 (e.g., a system bus and a memory bus) may provide electronic communication between the processor 212 and the memory, drives, interfaces, and other components of the machine 210. The machine 210 may be a physical or virtual device, such as a virtual router running on a hypervisor or within a container.

[0058] The processor 212, which may also be referred to as a central processing unit (“CPU”), may include any general-purpose or special-purpose processor capable of executing machine-readable instructions and performing operations on data in accordance with the instructions of the machine-readable instructions. The main memory 213 may be directly accessible by the processor 212 to access machine instructions and may be in the form of random access memory (“RAM”) or any type of dynamic memory (e.g., dynamic random access memory (“DRAM”)). The auxiliary memory 214 may be any non-volatile memory, such as a hard disk, capable of storing electronic data including executable software files. Electronically stored data from external storage may be provided to the computer 210 through one or more removable media drives 218, which may be configured to receive any type of external media, such as a compact disc (“CD”), a digital video disc (“DVD”), a flash drive, an external hard drive, etc.

[0059] Wireless, wired, and virtual network interfaces 215, 216A, and 216B can be provided to enable electronic communication via a network between machine 210 and other machines or nodes. In one example, wireless network interface 215 can include a wireless network controller (“WNIC”) having suitable transmit and receive components, such as transceivers, for wireless communication within the network. Wired network interface 216A can enable machine 210 to physically connect to the network via a wired line such as an Ethernet cable. Both wireless and wired network interfaces 215 and 216A can be configured to facilitate communication using a suitable communication protocol (e.g., Internet Protocol Suite (“TCP / IP”)). For illustrative purposes only, machine 210 is shown as having wireless and wired network interfaces 215 and 216A. Although one or more wireless and hardwired interfaces can be provided in machine 210 or externally connected to machine 210, only one connection option is required to enable machine 210 to connect to the network.

[0060] A user interface 217 can be provided in some machines to allow a user to interact with machine 210. User interface 217 can include a display device, such as a graphical display device (e.g., plasma display panel (“PDP”), liquid crystal display (“LCD”), cathode ray tube (“CRT”), etc.). Additionally, any suitable input mechanism can be included, such as a keyboard, touch screen, mouse, trackball, voice recognition, touchpad, and application programming interface (API), etc.

[0061] Removable media drive 218 represents a drive configured to receive any type of external computer-readable medium (e.g., computer-readable medium 219). Instructions embodying the activities or functions described herein can be stored on one or more external computer-readable media. Additionally, such instructions can also or alternatively reside at least partially within a storage element of machine 210 during execution (e.g., in main memory 213 or cache memory of processor 212), or within a non-volatile storage element of machine 210 (e.g., within secondary storage 214). Accordingly, other storage elements of machine 210 also constitute computer-readable media. Thus, “computer-readable media” is intended to include any medium that can store instructions executable by machine 210 that cause the machine to perform any one or more of the activities disclosed herein.

[0062] Figure 9No additional hardware is shown, which may be suitably coupled to the processor 212 and other components in the form of a memory management unit (“MMU”), additional symmetric multiprocessing elements, physical memory, a Peripheral Component Interconnect (“PCI”) bus and corresponding bridges, Small Computer System Interface (“SCSI”) / Integrated Drive Electronics (“IDE”) elements, etc. Machine 210 may include any additional suitable hardware, software, components, modules, interfaces, or objects that facilitate its operation. This may include suitable algorithms and communication protocols that allow for effective protection and communication of data. Additionally, any suitable operating system may be configured in machine 210 to suitably manage the operation of the hardware components therein.

[0063] The elements shown and / or described with reference to machine 210 are intended for illustrative purposes and are not meant to imply architectural limitations of the machine, such as those used in accordance with this disclosure. Additionally, each machine may include more or fewer components, and may operate as a virtual machine or virtual device, as appropriate and based on specific needs. As used herein in this specification, the term “machine” is intended to encompass any computing device or network element, such as a server, virtual server, logical container, router, personal computer, client computer, network device, switch, bridge, gateway, processor, load balancer, wireless LAN controller, firewall, or any other suitable device, component, element, or object operable to affect or process electronic information in a network environment.

[0064] In one example implementation, certain network elements or computing devices may be implemented as physical and / or virtual devices, and may include any suitable hardware, software, components, modules, or objects that facilitate their operation, as well as suitable interfaces for receiving, sending, and / or otherwise conveying data or information in a network environment. This may include suitable algorithms and communication protocols that allow for effective exchange of data or information.

[0065] Furthermore, in the embodiments described and shown herein, some of the processors and storage elements associated with various network elements may be removed or otherwise combined such that a single processor and a single memory location are responsible for certain activities. Alternatively, certain processing functions may be separated and separate processors and / or physical machines may implement the various functions. Generally, the arrangements depicted in the figures may be more logical in their representation, while the physical architecture may include various arrangements, combinations, and / or mixtures of these elements. It must be noted that countless possible design configurations may be used to achieve the operational objectives outlined herein. Thus, there are countless alternative arrangements, design choices, device possibilities, hardware configurations, software implementations, device options, etc. for the relevant infrastructure.

[0066] In some example embodiments, one or more memories may store data for the various operations described herein. This includes at least some memory elements capable of storing instructions (e.g., software, logic, code, etc.) that are executed to perform the activities described in this specification. A processor may execute any type of instructions associated with the data to implement the operations detailed in this specification. In one example, one or more processors may transform an element or article (e.g., data) from one state or thing to another. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor), and the elements identified herein may be some type of programmable processor, programmable digital logic (e.g., field programmable gate array (“FPGA”), erasable programmable read only memory (“EPROM”), electrically erasable programmable read only memory (“EEPROM”)), ASICs that incorporate digital logic, software, code, electronic instructions, flash memory, optical disks, CD-ROMs, DVD ROMs, magnetic or optical cards, other types of machine-readable media suitable for storing electronic instructions, or any suitable combination thereof.

[0067] Components of the environments shown herein may, where appropriate and based on particular needs, save information in any suitable type of memory (e.g., random access memory (“RAM”), read only memory (“ROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), etc.), software, hardware, or any other suitable component, device, element, or object. Any memory item discussed herein should be construed as being included within the broad term “storage element.” Information being read, used, tracked, sent, transmitted, communicated, or received by the network environments described herein may be provided in any database, register, queue, table, cache, control list, or other storage structure, all of which may be referenced within any suitable time frame. Any such storage option may be included within the broad term “storage element” used herein. Similarly, any potential processing elements and modules described in this specification should be construed as being included within the broad term “processor.”

[0068] In summary, in one embodiment, a printed circuit board (PCB) has a first central processing unit (CPU) socket and a second CPU socket that is substantially aligned with the first CPU socket, and also has a first plurality of dual in-line memory module (DIMM) sockets interconnected with the first CPU socket and a second plurality of DIMM sockets (in a direction parallel to the first plurality of DIMM sockets) interconnected with the second CPU socket. The first plurality of DIMM sockets are arranged in at least a first column and a second column of DIMM sockets on the PCB, and the second plurality of DIMM sockets are arranged in at least a second column and a third column of DIMM sockets on the PCB such that the second column of DIMM sockets contains interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets.

[0069] Note that, through the numerous examples provided herein, interactions can be described in terms of two, three, four, or more network elements. However, this is done for clarity and illustration only. It should be understood that the system can be incorporated in any suitable manner. Along similar design alternatives, any of the computers, modules, components, and elements shown in the figures can be combined in various possible configurations, all of which are clearly within the broad scope of this specification. In some cases, it may be easier to describe one or more functions of a given set of flows by referring to only a limited number of network elements. It should be understood that the embodiments described herein and their teachings, as shown in the figures, are readily extensible and can accommodate a large number of components as well as more complex / sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or inhibit the broad teachings of the system, as it may be applied to numerous other architectures.

[0070] It is also important to note that the operations and steps described with reference to the previous figures illustrate only some of the possible scenarios that can be performed by or within the system. Some of these operations can be appropriately deleted or removed, or the steps can be significantly modified or changed, without departing from the scope of the concepts being discussed. Additionally, the timing of these operations can be significantly altered and still achieve the results taught in this disclosure. The previous operational flow is provided for purposes of example and discussion. The system provides a fair amount of flexibility, as any suitable arrangement, chronological order, configuration, and timing mechanism can be provided without departing from the teachings of the concepts being discussed.

[0071] In the foregoing description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to one of ordinary skill in the art that the disclosed embodiments may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the disclosed embodiments. Additionally, references in the specification to "one embodiment", "example embodiment", "embodiment", "another embodiment", "some embodiments", "various embodiments", "other embodiments", "alternative embodiments", etc. are intended to indicate that any feature associated with such an embodiment (e.g., elements, structures, modules, components, steps, operations, characteristics, etc.) is included in one or more of the embodiments of the present disclosure.

[0072] Although illustrative embodiments providing a multi-column staggered DIMM layout and routing topology have been shown and described, it should be understood that various other adaptations and modifications can be made within the spirit and scope of the embodiments herein. For example, while certain embodiments are described herein with respect to the use of certain types of memory or memory protocols, the embodiments herein are not limited thereto and can be used with other types of memory in other embodiments. Additionally, while certain processing or processor protocols or terms may have been shown, described, or implied herein, other suitable protocols or terms can be used accordingly.

[0073] The foregoing description is directed to particular embodiments. However, it will be apparent that other changes and modifications can be made to the described embodiments to achieve some or all of their advantages. Accordingly, the description is presented by way of example rather than limitation of the scope of the embodiments herein. Thus, the purpose of the appended claims is to cover all such changes and modifications that fall within the true spirit and scope of the embodiments herein.

Claims

1. A symmetric multiprocessing device, comprising: A printed circuit board (PCB); A first central processing unit (CPU) socket located on the PCB; A second CPU socket located on the PCB and substantially aligned with the first CPU socket on the PCB; A first plurality of dual in-line memory module (DIMM) sockets located on the PCB and interconnected with the first CPU socket, the first plurality of DIMM sockets being located on opposite first and second sides of the first CPU socket; And A second plurality of DIMM sockets located on the PCB and interconnected with the second CPU socket, the second plurality of DIMM sockets being located on opposite first and second sides of the second CPU socket and in a direction parallel to the first plurality of DIMM sockets; Wherein, the first plurality of DIMM sockets are arranged in at least a first column of DIMM sockets and a second column of DIMM sockets on the PCB across both the opposite first and second sides of the first CPU socket; and Wherein, the second plurality of DIMM sockets are arranged in at least the second column of DIMM sockets and a third column of DIMM sockets on the PCB across both the opposite first and second sides of the second CPU socket, such that the second column of DIMM sockets contains interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets, Wherein, the second column of DIMM sockets containing interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets is disposed between the first column and the third column.

2. The symmetric multiprocessing device according to claim 1, wherein: The first column and the second column are separated by the center line of the first CPU socket; and The second column and the third column are separated by the center line of the second CPU socket.

3. The symmetric multiprocessing device according to claim 2, wherein: The portion of the first plurality of DIMM sockets closest to the first CPU socket is further disposed on the center line of the first CPU socket on the PCB; and The portion of the second plurality of DIMM sockets closest to the second CPU socket is further disposed on the center line of the second CPU socket on the PCB.

4. The symmetric multiprocessing device according to claim 3, wherein: The portion of the first plurality of DIMM sockets closest to the first CPU socket includes two DIMM sockets on the first side of the first CPU socket and two DIMM sockets on the second side of the first CPU socket; and The portion of the second plurality of DIMM sockets closest to the second CPU socket includes two DIMM sockets on the first side of the second CPU socket and two DIMM sockets on the second side of the second CPU socket.

5. The symmetric multiprocessing device according to any one of claims 1 to 4, wherein: The first plurality of DIMM sockets in the first column include four DIMM sockets on the first side of the first CPU socket and four DIMM sockets on the second side of the first CPU socket; The second plurality of DIMM sockets in the third column include four DIMM sockets on the first side of the second CPU socket and four DIMM sockets on the second side of the second CPU socket; The first plurality of DIMM sockets in the second column include two DIMM sockets on the first side of the first CPU socket and two DIMM sockets on the second side of the first CPU socket, and The second plurality of DIMM sockets in the second column include two DIMM sockets on the first side of the second CPU socket and two DIMM sockets on the second side of the second CPU socket.

6. The symmetric multi - processing device according to claim 1, further comprising: A memory channel interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets; Wherein, the pins of the first CPU socket and the second CPU socket are rotated on the PCB to the following direction: among the four possible rotation directions, this direction provides the shortest length for the longest memory channel in the memory channel.

7. The symmetric multi - processing device according to claim 1, further comprising: A memory channel interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets; Wherein, the longest memory channel in the memory channel is in the range between 2.0 and 7.0 inches.

8. The symmetric multiprocessing device according to claim 1, wherein, The PCB has a thickness less than 140 mils.

9. The symmetric multiprocessing device according to claim 1, wherein, The first plurality of DIMM sockets and the second plurality of DIMM sockets are configured for one of double - data - rate (DDR) fourth - generation (DDR4) DIMMs or DDR fifth - generation (DDR5) DIMMs.

10. The symmetric multiprocessing device according to claim 1, wherein, The device is a network device.

11. A printed circuit board (PCB), comprising: A first central processing unit (CPU) socket; A second CPU socket, substantially aligned with the first CPU socket; A first plurality of dual - in - line memory module (DIMM) sockets, interconnected with the first CPU socket, the first plurality of DIMM sockets being located on opposite first and second sides of the first CPU socket; And A second plurality of DIMM sockets, interconnected with the second CPU socket, the second plurality of DIMM sockets being located on opposite first and second sides of the second CPU socket and in a direction parallel to the first plurality of DIMM sockets; Wherein, the first plurality of DIMM sockets are arranged in at least a first column of DIMM sockets and a second column of DIMM sockets on the PCB across both the opposite first and second sides of the first CPU socket; and Among them, the second plurality of DIMM sockets are arranged in at least the second column DIMM sockets and the third column DIMM sockets on the PCB across both the opposite first side and the second side of the second CPU socket, such that the second column DIMM sockets include interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets. Among them, the second column DIMM sockets including interleaved DIMM sockets from each of the first plurality of DIMM sockets and the second plurality of DIMM sockets are arranged between the first column and the third column.

12. The PCB according to claim 11, wherein: The first column and the second column are separated by the center line of the first CPU socket; and The second column and the third column are separated by the center line of the second CPU socket.

13. The PCB according to claim 12, wherein: The portion of the first plurality of DIMM sockets closest to the first CPU socket is further arranged on the center line of the first CPU socket; and The portion of the second plurality of DIMM sockets closest to the second CPU socket is further arranged on the center line of the second CPU socket.

14. The PCB according to claim 13, wherein: The portion of the first plurality of DIMM sockets closest to the first CPU socket includes two DIMM sockets on the first side of the first CPU socket and two DIMM sockets on the second side of the first CPU socket; and The portion of the second plurality of DIMM sockets closest to the second CPU socket includes two DIMM sockets on the first side of the second CPU socket and two DIMM sockets on the second side of the second CPU socket.

15. The PCB according to any one of claims 11 to 14, wherein: The first plurality of DIMM sockets in the first column include four DIMM sockets on the first side of the first CPU socket and four DIMM sockets on the second side of the first CPU socket; The second plurality of DIMM sockets in the third column include four DIMM sockets on the first side of the second CPU socket and four DIMM sockets on the second side of the second CPU socket; The first plurality of DIMM sockets in the second column include two DIMM sockets on the first side of the first CPU socket and two DIMM sockets on the second side of the first CPU socket, and The second plurality of DIMM sockets in the second column include two DIMM sockets on the first side of the second CPU socket and two DIMM sockets on the second side of the second CPU socket.

16. The PCB according to claim 11, further comprising: A memory channel interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets; wherein, the pins of the first CPU socket and the second CPU socket are rotated on the PCB to the following direction: among the four possible rotation directions, the direction provides the shortest length for the longest memory channel located in the memory channel.

17. The PCB according to claim 11, further comprising: A memory channel interconnecting the first CPU socket and the second CPU socket to the first plurality of DIMM sockets and the second plurality of DIMM sockets; wherein, the longest memory channel in the memory channel is less than or equal to 6.8 inches.

18. The PCB according to claim 11, wherein, The PCB has a thickness less than 135 mm.

19. The PCB according to claim 11, wherein, The first plurality of DIMM sockets and the second plurality of DIMM sockets are configured for one of double data rate (DDR) fourth generation (DDR4) DIMMs or DDR fifth generation (DDR5) DIMMs.

20. The PCB according to claim 11, wherein, The PCB is configured for a network device.

Images