Data center

Abstract

The invention discloses a data center, and belongs to the technical field of data center process layout. According to the scheme, a cluster unit of the data center is of a three-layer annular structure and is divided into a plurality of modules in the radial direction; a spine switch is deployed on the first annular layer, GPU server cabinets are deployed on the second annular layer and the third annular layer, and a leaf switch is further deployed on the second annular layer; the distances between the layers and the positions of the spine switches and the leaf switches are reasonably set, so that the distances between the GPU servers and the leaf switches in the modules, between the leaf switches and the spine switches and between the spine switches in the cluster units are all smaller than 5 meters, the GPU servers and the leaf switches are connected through copper cables, the leaf switches and the spine switches are connected through copper cables, and the spine switches are connected through copper cables. According to the scheme, through reasonable layout, the reliability of the network is improved, and the network delay, the networking cost and the power consumption are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data center process layout, and in particular to a data center. Background Art

[0002] With the rapid development of high-performance computing, artificial intelligence, especially generative artificial intelligence and large-scale pre-trained models, mankind is moving rapidly towards general artificial intelligence. The current demand for artificial intelligence computing power is growing, and the layout of traditional data centers is increasingly unable to meet the requirements of high-performance computing. The core technology of current AI computing power is to imitate the connection method of the human brain neural network. By stacking simple GPU (Graphics Processing Unit) cores through efficient packaging technology (such as CoWoS packaging technology), and finally packaging them on a substrate, a 2.5D or 3D form is formed to achieve efficient interconnection from GPU to HBM (High Bandwidth Memory). In addition, high-speed interconnection between GPUs inside the server is achieved through NVLink technology, and high-speed interconnection of each GPU between servers is achieved based on a lossless network based on RDMA (Remote Direct Memory Access) technology. Such a hardware environment requires that any GPU can achieve high-speed and lossless interconnection with all other GPUs, which poses a great challenge to the organizational design of the network.

[0003] At present, the layout of data centers is usually a linear column layout. This layout has been used for a long time and widely. Its main design consideration is the space utilization of the computer room. Since the computer room of the traditional data center is a rectangular structure, the cabinet power density is low, usually only 3-5-7 kilowatts. In addition, the business model is mainly based on north-south traffic, and the east-west traffic is very small and insensitive to latency, which makes it difficult to meet the communication requirements of large bandwidth and low latency in the east-west direction in the current computing power scenario. Due to the long distance between the server cabinet and the switch, the traditional data center is only suitable for the networking mode based on optical fiber transmission. The network interface of each GPU server requires an optoelectronic conversion module to convert the electrical signal in the GPU server into a laser signal, transmit it to the switch port, and then convert the laser signal back to an electrical signal through the optoelectronic conversion module for exchange and distribution. Subsequently, it is converted from the switch port to a laser signal again through the optoelectronic module, transmitted to the network interface of another GPU server, and converted again into an electrical signal and handed over to the GPU server for processing. The structure of the optoelectronic conversion module is very complex, and the reliability of the link is also affected by the number of optoelectronic conversions. For example, in NVIDIA's networking solution based on a 64-port InfiniBand switch, a leaf layer switch is used to implement networking of less than 32 H100 GPU servers. Four optoelectronic modules are required to perform continuous conversion (GPU-Leaf-GPU) to complete the routing of a single link. When there are more than 32 GPU servers and less than 256 GPU servers, a maximum of eight optoelectronic conversion modules are required to perform continuous conversion (GPU-Leaf-Spine-Leaf-GPU) in the routing of a single link. At this time, the reliability of a single optical transmission link is reduced by nearly 10 times. When there are more than 256 GPU servers, a third core layer network needs to be added. At this time, a single optical transmission link requires 12 optoelectronic conversions (GPU-Leaf-Spine-Core-Spine-Leaf-GPU), and the reliability of a single link is further reduced. During the training of a large model, any optoelectronic conversion failure may lead to congestion of training data and an increase in system failure rate, thereby affecting training efficiency and significantly increasing costs.

[0004] In addition, each optoelectronic conversion module has a delay of 30-40 nanoseconds. Coupled with the long-distance optical fiber link and a large number of optoelectronic conversion modules, the system delay affects the high-frequency and high-flow data throughput efficiency of large model training, prolonging the training time of large models and increasing costs. In addition, the large number of optoelectronic conversion modules in the network leads to high networking costs and high power consumption. Summary of the invention

[0005] The technical problem to be solved by the present invention is: in view of the above-mentioned defects of the prior art, a data center is provided to solve the problems of reliability, high delay, high cost, high power consumption, etc. caused by the large-scale use of optoelectronic conversion modules in the layout of traditional data centers.

[0006] To achieve the above object, the present invention provides a data center, which is networked based on the networking solution of NVIDIA's 64-port Infiniband switch, characterized in that it includes a single-story or multi-story building, and each floor of the building includes at least one cluster unit;

[0007] The cluster unit is a three-layer ring structure, which includes the first ring layer RL1, the second ring layer RL2 and the third ring layer RL3 from the inside to the outside;

[0008] The cluster unit is divided into A modules along the radial direction; the first ring layer RL1 in each module is provided with a network cabinet 1, and the network cabinet 1 is provided with B spine switches; the second ring layer RL2 and the third ring layer RL3 in each module are provided with C server cabinets 2, and each server cabinet 2 is provided with D GPU servers; the top layer of the server cabinet 2 of the second ring layer RL2 of each module is provided with E leaf switches;

[0009] Each of the GPU servers is equipped with multiple GPU cards, and the GPU cards of each of the GPU servers support low-latency interconnection communication;

[0010] There is a first distance L1 between the outer edge of the first ring layer RL1 and the inner edge of the second ring layer RL2; there is a second distance L2 between the outer edge of the second ring layer RL2 and the inner edge of the third ring layer RL3; by reasonably setting the first distance L1 and the second distance L2, as well as the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, between the leaf switch and the spine switch, and between the spine switches in the cluster unit are all less than 5 meters; the GPU server in each module is connected to the leaf switch in the same module through a copper cable; the leaf switch in each module is connected to the spine switch in the same module through a copper cable; the spine switches in the cluster unit are connected through copper cables;

[0011] A core switch is also provided in each of the network cabinets 1 , the spine switch is connected to the core switch via a copper cable, and the cluster units are interconnected via the core switch.

[0012] Preferably, installation and maintenance channels are reserved in the middle of the first annular layer RL1, the second annular layer RL2, and the third annular layer RL3.

[0013] Preferably, the spine switch and the leaf switch each include 64 ports, wherein the 32 downstream ports of the leaf switch are used to connect to the GPU server, and the 32 upstream ports are used to connect to the spine switch; the 32 downstream ports of the spine switch are used to connect to the leaf switch, and the 32 upstream ports are used to connect to other spine switches.

[0014] Preferably, A=8, B=8, C=4, D=4, E=2, each of the GPU servers is equipped with 8 GPU cards; the first distance L1 and the second distance L2 are both 1.2 meters.

[0015] Preferably, the core switches of multiple cluster units on the same floor are connected by optical cables; when the distance between the core switches of two cluster units on adjacent floors is less than 5 meters, copper cables are used to connect the core switches of the two cluster units, otherwise, optical cables are used to connect the core switches of the two cluster units.

[0016] Preferably, the model of the GPU card in the GPU server is NVIDIA H100.

[0017] Preferably, the model of the GPU card in the GPU server is NVIDIA H200.

[0018] Preferably, wire mesh is provided between the top of the first annular layer RL1 and the top of the second annular layer RL2, and between the top of the second annular layer RL2 and the top of the third annular layer RL3, and the copper cables connecting the adjacent annular layers are fixed and supported by the wire mesh.

[0019] Preferably, the server cabinets 2 in the second ring layer RL2 and the third ring layer RL3 of each module are arranged into a server cabinet row along the edge of the ring layer, and a power distribution unit 3 is provided at the head of the server cabinet queue, and the power distribution unit 3 is connected to the server cabinet 2 via a cable.

[0020] Preferably, the head or tail of the server cabinet queue of the third ring layer RL3 of each module is equipped with a cooling and dehumidification unit 4, the GPU chip and the CPU chip in the GPU server are fixed to the water cooling plate through thermal conductive glue, and the water cooling plate is connected to the cooling and dehumidification unit 4 through a stainless steel water cooling pipe; the network cabinet 1, the server cabinet 2, and the power distribution unit 3 are all provided with a cabinet door 5, and a chilled water fan coil cabinet panel 6 is provided on the outside of the cabinet body on the side opposite to the cabinet door 5.

[0021] The present invention has the following beneficial effects: the solution of the present invention innovatively adopts a three-layer ring structure layout, and each cluster unit is a three-layer ring structure, which is the first ring layer, the second ring layer, and the third ring layer from the inside to the outside. The cluster unit is divided into multiple modules along the radial direction; the first ring layer of each module is provided with a network cabinet, and a spine switch is provided in the network cabinet; the second ring layer and the third ring layer of each module are provided with a server cabinet, and each server cabinet is provided with multiple GPU servers; the top layer of the server cabinet of the second ring layer of each module is provided with a leaf switch; by reasonably setting the distance between each ring layer and carefully arranging the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, the distance between the leaf switch and the spine switch, and the distance between the spine switches in the cluster unit are all less than 5 meters, and the GPU server and the leaf switch, the leaf switch and the spine switch, and the spine switches are all connected by copper cables. This solution reduces the distance between components in the network through reasonable layout and uses copper cables instead of optical cables, which improves network reliability, reduces network latency, networking costs and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 A schematic diagram of the layout structure of a cluster unit consisting of 256 GPU servers.

[0024] Figure 2 A schematic diagram of the network topology of a cluster unit consisting of 256 GPU servers.

[0025] Figure 3 A schematic diagram of the layout structure of a cluster unit consisting of 32 GPU servers.

[0026] Figure 4 A schematic diagram of the layout structure of a cluster unit consisting of 128 GPU servers. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The general idea of ​​the present invention is that each cluster unit is a three-layer ring structure, which is the first ring layer, the second ring layer, and the third ring layer from the inside to the outside. The cluster unit is divided into multiple modules along the radial direction; the first ring layer of each module is deployed with a spine switch, the second ring layer and the third ring layer are both deployed with a GPU server cabinet, and the second ring layer is also deployed with a leaf switch; by reasonably setting the distance between each layer and carefully arranging the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, the leaf switch and the spine switch, and the spine switch in the cluster unit are all less than 5 meters, and the GPU server and the leaf switch, the leaf switch and the spine switch, and the spine switches are all connected by copper cables.

[0029] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings of the specification. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0030] like Figure 1 As shown, an embodiment of the present invention provides a data center, which is networked based on the networking solution of NVIDIA's 64-port Infiniband switch. The data center includes a single-story or multi-story building, and each floor of the building includes at least one cluster unit; the cluster unit is a three-layer ring structure, which includes a first ring layer RL1, a second ring layer RL2 and a third ring layer RL3 from the inside to the outside.

[0031] The cluster unit is radially divided into A modules; the first ring layer RL1 in each module is provided with a network cabinet 1, and B spine switches are provided in the network cabinet 1; the second ring layer RL2 and the third ring layer RL3 in each module are provided with C server cabinets 2, and each server cabinet 2 is provided with D GPU servers; the top layer of the server cabinet 2 of the second ring layer RL2 of each module is provided with E leaf switches.

[0032] The structure of a single module is as follows Figure 3 shown. Figure 1There are 8 modules in total, corresponding to M1-M8 in the figure.

[0033] Each of the GPU servers is equipped with multiple GPU cards, and the GPU cards of each GPU server support low-latency interconnection communication.

[0034] There is a first distance L1 between the outer edge of the first ring layer RL1 and the inner edge of the second ring layer RL2; there is a second distance L2 between the outer edge of the second ring layer RL2 and the inner edge of the third ring layer RL3; by reasonably setting the first distance L1 and the second distance L2, as well as the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, between the leaf switch and the spine switch, and between the spine switches in the cluster unit are all less than 5 meters; the GPU server in each module is connected to the leaf switch in the same module through a copper cable; the leaf switch in each module is connected to the spine switch in the same module through a copper cable; the spine switches in the cluster unit are connected through copper cables. In actual applications, several fixed copper cable specifications can be unified according to the distance from the GPU server to the leaf switch, the distance from the leaf switch to the spine switch, and the distance between the spine switches. For example, copper cables of 2 meters, 3 meters, and 4 meters in length can be selected according to the actual distance to improve cable consistency and enhance the accuracy of topology perception.

[0035] A core switch is also provided in each of the network cabinets 1 , the spine switch is connected to the core switch via a copper cable, and the cluster units are interconnected via the core switch.

[0036] In the embodiment of the present invention, installation and maintenance channels are reserved in the middle of the first annular layer RL1, the second annular layer RL2, and the third annular layer RL3. Figure 1 As shown, the cluster unit is a left-right symmetrical structure, with 4 modules on each side. Installation and maintenance channels are reserved between the modules M1 and M8, and between the modules M4 and M5. The installation and maintenance channels run through the first annular layer RL1, the second annular layer RL2, and the third annular layer RL3.

[0037] In an embodiment of the present invention, the spine switch and the leaf switch each include 64 ports, wherein the 32 downstream ports of the leaf switch are used to connect to the GPU server, and the 32 upstream ports are used to connect to the spine switch; the 32 downstream ports of the spine switch are used to connect to the leaf switch, and the 32 upstream ports are used to connect to other spine switches.

[0038] like Figure 1 As shown, in some embodiments of the present invention, A=8, B=8, C=4, D=4, E=2, and each of the GPU servers is equipped with 8 GPU cards. Figure 1 In the cluster unit, there are 8 modules in total, each module forms a fan-shaped structure along the radial direction of the cluster unit, and the second ring layer RL2 and the third ring layer RL3 of each module form an approximately octagonal structure. The first ring layer RL1 in each module is equipped with 8 64-port spine switches, the second ring layer RL2 and the third ring layer RL3 are each equipped with 4 server cabinets 2, and each server cabinet 2 is equipped with 4 GPU servers. Each server cabinet 2 of the second ring layer RL2 is equipped with 2 64-port leaf switches, and each GPU server is equipped with 8 GPU cards. Therefore, Figure 1 In the illustrated embodiment, each module has 8x4=32 GPU servers, and a cluster unit has a total of 8x32=256 GPU servers. Each module has 32x8 GPU cards, which requires 8 64-port leaf switches and 8 64-port spine switches. The 32 downstream ports of the leaf switch are used to connect to the GPU servers, and the 32 upstream ports are used to connect to the spine switch; the 32 downstream ports of the spine switch are used to connect to the leaf switch, and the 32 upstream ports are used to connect to the other spine switches. Figure 2 for Figure 1 Network topology diagram of the structure shown.

[0039] Figure 1 In the embodiment, two network cabinets 1 are arranged in the first ring layer of each module, and four 64-port spine switches are arranged on the top layer of each network cabinet 1. In practical applications, the number of network cabinets 1 and the number of spine switches in each network cabinet 1 can be set as needed.

[0040] In actual applications, if the GPU server is equipped with 4 GPU cards, 16 GPU cards, or a combination of GPU servers with multiple numbers of GPU cards, the number of leaf switches, spine switches, server cabinets, and servers in the server cabinets can be adjusted according to actual conditions.

[0041] In the embodiment of the present invention, when the number of GPU servers is less than 256, 32 GPU servers are still arranged as one module. Figure 3 As shown in the figure, it is a schematic diagram of the layout of 32 GPU servers, and 32 GPU servers constitute 1 module. Figure 4 As shown, it is a schematic diagram of the layout of 128 GPU servers. The 128 GPU servers constitute 4 modules. In order to facilitate subsequent expansion, the 4 modules are arranged continuously.

[0042] In an embodiment of the present invention, the length, width and height of the server cabinet are 1.2 meters, 0.6 meters and 2-2.5 meters respectively, and 4 GPU servers with a height of 8U-10U can be placed in each server cabinet, where U is the basic unit for measuring the height of the equipment in the server rack, and 1U≈44.45mm. The model of the GPU card in the GPU server is NVIDIA H100 or NVIDIA H200. NVIDIAH100 is a high-performance GPU accelerator launched by NVIDIA, designed for data centers and high-performance computing (HPC). H100 is based on the NVIDIA Hopper architecture and supports NVLink high-speed interconnection technology. It can achieve high-speed communication between multiple GPUs and is suitable for high-performance computing, deep learning training, deep learning reasoning, and data center acceleration. NVIDIA H200 is the next generation product of NVIDIA H100 and has the same size and power consumption as NVIDIA H100. It should be noted that the GPU cards supported by the present invention are not limited to NVIDIA H100 and NVIDIA H200. In practical applications, the size of the server cabinet can be adjusted according to the model of the GPU card and the size of the GPU server.

[0043] In the embodiment of the present invention, the first distance L1 and the second distance L2 are both 1.2 meters. A leaf switch is set on the top layer of the server cabinet 2 of the second ring layer RL2, and a spine switch is set on the top layer of the network cabinet 1 of the first ring layer RL1. In this way, it is ensured that the distance between the GPU server and the leaf switch in each module, the distance between the leaf switch and the spine switch, and the distance between the spine switches in the cluster unit are all less than 5 meters, and the GPU server and the leaf switch, the leaf switch and the spine switch, and the spine switches can all be connected by copper cables.

[0044] Copper cables have a lower bit error rate than other materials, which helps to improve the reliability of data transmission and reduce the error rate during data transmission. Especially in large-scale clusters, stable data transmission is crucial to the stability of the entire system. In the embodiments of the present invention, passive direct-connect cables (DAC, <3m) and active cables (AEC, <5m) are used to achieve direct connection of all-electrical signals. Compared with optical cables, DAC / AEC copper cables operate at lower temperatures, consume less power, have lower costs, and are more reliable. Therefore, this design reduces intermittent paralysis and failures of network links, which is the main problem faced by all high-speed interconnects using optical devices. By Figure 1 In the layout shown, the delay difference between each downstream port of the leaf switch and the delay difference between each downstream port of the spine switch are controlled within a very small range, the distance difference between the spine switches is also minimized as much as possible, and the delay difference of each link in the network is greatly reduced. In the solution of the embodiment of the present invention, the cable length and the communication route maintain a positive relationship, which avoids the negative impact of communication delay on topology perception and improves the accuracy of topology perception. Compared with the use of optical cable connection, the total length of the cable in the network is greatly reduced, the signal quality is significantly improved, and the signal round-trip time is greatly reduced, which significantly improves the interconnection efficiency between GPU servers and reduces the networking cost.

[0045] In actual applications, the first distance L1, the second distance L2, and the positions of leaf switches, spine switches, and server cabinets can be adjusted according to the specific conditions of the data center and industry specifications, so that the distance between the GPU server and the leaf switch in each module, the distance between the leaf switch and the spine switch, and the distance between the spine switches are all less than 5 meters or as less than 5 meters as possible, so as to minimize the number of optoelectronic conversion modules in the network.

[0046] In some embodiments of the present invention, wire mesh is provided between the top of the first ring layer RL1 and the top of the second ring layer RL2, and between the top of the second ring layer RL2 and the top of the third ring layer RL3, and the copper cables connected between adjacent ring layers are fixed and load-bearing through the wire mesh. The wire mesh is provided to arrange the cables neatly and orderly, which is convenient for subsequent maintenance. In other embodiments of the present invention, a bridge corresponding to the positions of the first ring layer RL1, the second ring layer RL2, and the third ring layer RL3 is provided above the cluster unit to support and organize the copper cables.

[0047] In the embodiment of the present invention, when the number of GPU servers is greater than 256, see Figure 1Multiple clusters are built in the manner shown, and the core switches of multiple cluster units on the same floor are connected by optical cables; when the distance between the core switches of two cluster units on adjacent floors is less than 5 meters, copper cables are used to connect the core switches of the two cluster units, otherwise, optical cables are used to connect the core switches of the two cluster units. When cluster units are deployed on adjacent floors, the cluster units in two adjacent floors can be relatively arranged in a vertical direction. If the distance between the core switches of two cluster units is less than 5 meters, vertical wiring can be performed through a vertical shaft, a cable trough or a pipeline system, and copper cables can be used to connect the core switches of the two cluster units. Therefore, the solution of the present invention has good network scalability, and can easily achieve expansion from 256 servers to 512 servers and 1024 servers, and the distance between clusters is also greatly reduced, which minimizes system delays and improves reliability while providing flexibility and sustainability for future cluster growth, while ensuring performance stability during expansion.

[0048] like Figure 1 , Figure 3 , Figure 4 As shown, in the embodiment of the present invention, the server cabinets 2 in the second ring layer RL2 and the third ring layer RL3 of each module are arranged into a server cabinet row along the edge of the ring layer, and a power distribution unit 3 is provided at the head of the server cabinet row. The power distribution unit 3 is connected to the cabinets, switches and other equipment in the server cabinet 2 through cables, and each server cabinet 2 is equipped with two UPS power supplies.

[0049] In the embodiment of the present invention, multiple GPU servers are concentrated in one server cabinet. In order to extend the service life of the GPU chip, it is necessary to accurately dissipate heat for the GPU server. Figure 1 , Figure 3 , Figure 4 As shown, in the embodiment of the present invention, the head or tail of the server cabinet queue of the third ring layer RL3 of each module is equipped with a cooling and dehumidification unit 4, and the GPU chip and the CPU chip in the GPU server are fixedly attached to the water-cooled plate through thermal conductive glue, and the water-cooled plate is connected to the cooling and dehumidification unit 4 through a stainless steel water-cooling pipe. The network cabinet 1, the server cabinet 2, and the power distribution unit 3 are all provided with a cabinet door 5, and a chilled water fan coil cabinet panel 6 is provided on the outside of the cabinet body on the side opposite to the cabinet door 5. In the data center computer room, an air conditioner for automatic dehumidification is also installed. Through liquid cooling, chilled water fan coil, and air conditioning, the reliability of long-term operation of the GPU server is improved.

[0050] The present invention has the following beneficial effects: the solution of the present invention innovatively adopts a three-layer ring structure layout, and each cluster unit is a three-layer ring structure, which is the first ring layer, the second ring layer, and the third ring layer from the inside to the outside. The cluster unit is divided into multiple modules along the radial direction; the first ring layer of each module is provided with a network cabinet, and a spine switch is provided in the network cabinet; the second ring layer and the third ring layer of each module are provided with a server cabinet, and each server cabinet is provided with multiple GPU servers; the top layer of the server cabinet of the second ring layer of each module is provided with a leaf switch; by reasonably setting the distance between each ring layer and carefully arranging the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, the distance between the leaf switch and the spine switch, and the distance between the spine switches in the cluster unit are all less than 5 meters, and the GPU server and the leaf switch, the leaf switch and the spine switch, and the spine switches are all connected by copper cables. This solution reduces the distance between components in the network through reasonable layout and uses copper cables instead of optical cables, which improves network reliability, reduces network latency, networking costs and power consumption.

[0051] The above are only specific implementations of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A data center is networked based on a networking solution of a 64-port Infiniband switch of NVIDIA, characterized in that: It includes a single-story or multi-story building, each floor of the building includes at least one cluster unit; The cluster unit is a three-layer ring structure, which includes a first ring layer (RL1), a second ring layer (RL2) and a third ring layer (RL3) from the inside to the outside. The cluster unit is divided into A modules along the radial direction; the first ring layer (RL1) in each module is provided with a network cabinet (1), and B spine switches are provided in the network cabinet (1); the second ring layer (RL2) and the third ring layer (RL3) in each module are provided with C server cabinets (2), and D GPU servers are provided in each server cabinet (2); the top layer of the server cabinet (2) in the second ring layer (RL2) of each module is provided with E leaf switches; Each of the GPU servers is equipped with multiple GPU cards, and the GPU cards of each of the GPU servers support low-latency interconnection communication; There is a first distance (L1) between the outer edge of the first ring layer (RL1) and the inner edge of the second ring layer (RL2); there is a second distance (L2) between the outer edge of the second ring layer (RL2) and the inner edge of the third ring layer (RL3); by reasonably setting the first distance (L1) and the second distance (L2), as well as the positions of the spine switch and the leaf switch, the distance between the GPU server and the leaf switch in each module, between the leaf switch and the spine switch, and between the spine switches in the cluster unit are all less than 5 meters; the GPU server in each module is connected to the leaf switch in the same module through a copper cable; the leaf switch in each module is connected to the spine switch in the same module through a copper cable; the spine switches in the cluster unit are connected to each other through copper cables; Each network cabinet (1) is also provided with a core switch, the spine switch is connected to the core switch via a copper cable, and the cluster units are interconnected via the core switch.

2. The data center according to claim 1, characterized in that: Installation and maintenance channels are reserved in the middle of the first annular layer (RL1), the second annular layer (RL2), and the third annular layer (RL3).

3. The data center according to claim 1, characterized in that: The spine switch and the leaf switch each include 64 ports, wherein the 32 downstream ports of the leaf switch are used to connect to the GPU server, and the 32 upstream ports are used to connect to the spine switch; the 32 downstream ports of the spine switch are used to connect to the leaf switch, and the 32 upstream ports are used to connect to other spine switches.

4. The data center according to claim 3, characterized in that: A=8, B=8, C=4, D=4, E=2, each of the GPU servers is equipped with 8 GPU cards; the first distance (L1) and the second distance (L2) are both 1.2 meters.

5. The data center according to claim 1, characterized in that: The core switches of multiple cluster units on the same floor are connected by optical cables; when the distance between the core switches of two cluster units on adjacent floors is less than 5 meters, copper cables are used to connect the core switches of the two cluster units, otherwise, optical cables are used to connect the core switches of the two cluster units.

6. The data center according to claim 1, characterized in that: The model of the GPU card in the GPU server is NVIDIA H100.

7. The data center according to claim 1, characterized in that: The model of the GPU card in the GPU server is NVIDIA H200.

8. The data center according to claim 1, characterized in that: Wire meshes are arranged between the top of the first annular layer (RL1) and the top of the second annular layer (RL2), and between the top of the second annular layer (RL2) and the top of the third annular layer (RL3), and the copper cables connecting the adjacent annular layers are fixed and supported by the wire meshes.

9. The data center according to claim 1, characterized in that: The server cabinets (2) in the second annular layer (RL2) and the third annular layer (RL3) of each module are arranged into a server cabinet row along the edge of the annular layer, and a power distribution unit (3) is provided at the head of the server cabinet row, and the power distribution unit (3) is connected to the server cabinet (2) via a cable.

10. The data center according to claim 9, characterized in that: The head or tail of the server cabinet queue of the third annular layer (RL3) of each module is equipped with a cooling and dehumidifying unit (4); the GPU chip and the CPU chip in the GPU server are fixedly attached to a water cooling plate via a thermally conductive adhesive; the water cooling plate is connected to the cooling and dehumidifying unit (4) via a stainless steel water cooling pipe; the network cabinet (1), the server cabinet (2), and the power distribution unit (3) are all provided with a cabinet door (5); and a chilled water fan coil cabinet panel (6) is provided on the outer side of the cabinet body on the side opposite to the cabinet door (5).

Images