High-bandwidth computing architecture based on storage hierarchy reconstruction and data processing method thereof

Abstract

The application discloses a high-bandwidth, large-capacity and high-reliability computing architecture based on storage hierarchy reconstruction, a data processing method and system. It belongs to the field of computer architecture, high-performance computing (HPC) and artificial intelligence hardware acceleration. The architecture adopts an innovative design of "small node, large cluster", and is composed of a large cluster which can be infinitely expanded by standardized storage and computing integrated small nodes. The core innovation lies in: reconstructing the storage hierarchy, using SSD as main memory, and directly transmitting data between SSD and cache in the processor chip, eliminating multi-level data copying; and realizing strict matching of full-link bandwidth, so that GPU computing power and Ethernet bandwidth grow synchronously, ensuring that the bandwidth is aligned within the node, the chassis, the rack and across the data center. The application can make the computing power utilization rate and bandwidth utilization rate reach 100%; reduce the system construction cost to less than 1 / 10 of the traditional scheme, reduce the power consumption to 1 / 9, increase the single-node main memory capacity by 20 times to 340,000 times, and significantly shorten the large model training period.

Description

Technical Field

[0001] This invention relates to the fields of computer architecture, high-performance computing (HPC) and artificial intelligence hardware acceleration, and in particular to a high-bandwidth, high-capacity computing architecture based on storage hierarchy reconstruction and its data processing method. Background Technology

[0002] Existing computing architectures suffer from bandwidth issues due to the "memory wall," resulting in effective computing power far below theoretical peak performance. Taking the NVIDIA H100 GPU as an example, using the formula for computing power utilization: main memory bandwidth / (theoretical computing power × precision bytes × memory access instruction ratio), since the memory access instruction ratio equals 1 - cache hit rate, in applications like large model inference and pre-training where the cache hit rate is 0, the H100's memory access instruction ratio is 1. Therefore, computing power utilization = 3.5 / (1979 × 2 × 1) × 100% ≈ 0.08%, and effective computing power is only 1979 × 0.08% = 1.675 TFLOPS (FP16). To improve main memory bandwidth, main memory can only be made infinitely closer to the processor, creating a vicious cycle of decreasing main memory capacity and increasing cost.

[0003] Therefore, there is an urgent need in this field for a computing cluster architecture that can overcome bandwidth bottlenecks, significantly improve computing power utilization and main memory capacity, and reduce costs and power consumption.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a computing cluster architecture that can overcome bandwidth bottlenecks, significantly improve computing power utilization and main memory capacity, and reduce costs and power consumption.

[0006] To achieve the above objectives, the present invention provides the following solution: A high-bandwidth computing architecture based on storage tier reconstruction includes: Multiple standardized small nodes integrating storage and computing; Each of the aforementioned small nodes includes: The processor module includes a CPU, a GPU, and a multi-level cache; it includes a CPU with at least 4 cores and a GPU that supports double speed and can pack two FP16 operations into a single FP32 register for simultaneous execution. Main memory uses non-volatile SSD storage to directly store the data and model parameters required for computation; The interconnect interface has PCIe×16 bandwidth matching the computing power of the GPU; In this architecture, data is directly transferred between the SSD and the high-speed cache built into the processor chip, without needing to pass through the memory and video memory.

[0007] Optionally, the storage module of the small node includes at least four PCIe×4 NVMeSSDs of the latest PCIe version, and the aggregated bandwidth is not less than the PCIe×16 bandwidth through striping technology.

[0008] Optionally, the computing power of the GPU increases synchronously with the bandwidth of the PCIe version ×16 used by the interconnect interface; when using PCIe 5.0 ×16, the FP32 computing power of the GPU is not less than 0.075 TFLOPS; when using PCIe 6.0 ×16, the FP32 computing power of the GPU is not less than 0.15 TFLOPS.

[0009] Across data centers, interconnection is achieved using high-speed Ethernet adapted to the internal PCIe ×16 bandwidth of the small nodes.

[0010] Optionally, when the small node uses PCIe 5.0 ×16, the high-speed Ethernet across data centers is 400GbE; when the small node uses PCIe 6.0 ×16, the high-speed Ethernet across data centers is 800GbE.

[0011] A data processing method based on a high-bandwidth computing architecture with storage hierarchy reconstruction includes the following steps: Data loading steps: The working set is stored in the SSD main memory of each small node, and during computation, it is directly read from the SSD main memory to the processor's cache; Distributed computing steps: Multiple small nodes collaborate in computing through a hierarchical interconnection architecture. Each small node undertakes a portion of the computing tasks, and the nodes are interconnected through a high-speed network to transmit and store intermediate computing results. Result storage steps: The calculation results are directly written to the SSD main memory of this node, and the data is backed up only when a permanent backup is required.

[0012] Optionally, in the distributed computing step, it is not necessary to set checkpoints at fixed time intervals during the computing process to write intermediate computing results to non-volatile storage.

[0013] A high-bandwidth computing system based on storage hierarchy reconstruction includes: Multiple high-bandwidth computing architectures based on storage tier reconstruction as described above; and an interconnection network connecting the multiple computing architectures; In this method, the computing power and storage capacity can be infinitely expanded by increasing the number of small nodes.

[0014] Optionally, the computing power utilization rate of the system can reach 100%, and the system construction cost and operating power consumption are significantly reduced.

[0015] Optionally, when used for training large models, its training cycle is shortened by more than 30% compared to the training cycle using traditional GPU clusters.

[0016] Compared with the prior art, the present invention has the following beneficial effects: Significantly improved computing power utilization: Through the "in-memory computing" design and bandwidth adaptation optimization, GPU computing power and PCIe bandwidth grow in tandem. The bandwidth of each small node is always close to the theoretical value under the corresponding PCIe ×16 bandwidth limit, which greatly increases computing power utilization and completely solves the problem of computing power waste caused by the bandwidth bottleneck of the existing cluster. The computing power utilization is more than 1,000 times higher than the existing NVIDIA GPU multi-card cluster.

[0017] Significantly reduced cost and power consumption: Individual small nodes utilize low-cost hardware configurations, eliminating the need for high-specification GPUs and cooling / power supply equipment. The overall cluster construction cost is only 3.86% of the existing H1000-card cluster, with operating power consumption at only 12.15%, effective computing power at 71.6%, and main memory capacity 100 times greater. It can train large models with over ten trillion parameters, significantly lowering the barrier to entry for training and inference of large models; simultaneously, it supports PCIe version iteration upgrades without requiring a complete cluster reconstruction, further reducing upgrade costs.

[0018] Shorter training and inference cycles: Eliminating multi-level data migration between memory and GPU memory, and eliminating the need to set checkpoints to back up intermediate calculation results, not only reduces data transmission latency, but also shortens the training time of large models by more than 30% compared to existing technologies.

[0019] Extremely scalable: Adopting a standardized architecture of "small nodes, large clusters", it can achieve unlimited expansion of computing power and main memory capacity by increasing the number of small nodes, adapting to the training and inference needs of large models with different parameter scales, and its flexibility far exceeds that of existing cluster architectures; at the same time, it is compatible with the latest version of PCIe, enabling synchronous upgrades of bandwidth and computing power, and adapting to the future growth of computing power needs of large models.

[0020] Wide adaptability: It is suitable for both large model training and inference scenarios, solving the problems of separation of training and inference architecture and poor adaptability in existing technologies; it can maintain stable bandwidth and computing power performance in large model scenarios of various scales; the external high-speed Ethernet and PCIe ×16 bandwidth are compatible with the chassis, ensuring that there are no bottlenecks in cross-node and cross-data center communication. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a network connection diagram of a computing cluster system provided in an embodiment of the present invention.

[0023] Figure 2 This is a cabinet installation layout diagram provided for an embodiment of the present invention.

[0024] ① Chassis; ② Backbone switch; ③ Switch connection ports; ④ Top-mounted switch; ⑤ Data transmission cables.

[0025] Figure 3 This is a schematic diagram of a small node illustrating the separation of the data plane and control plane in an embodiment of the present invention.

[0026] ① 6-way PCIe 5.0 x4 cable; ② CPU / GPU and cache; ③ M.2 interface SSD (system disk); ④ 10GbE port; ⑤ Power supply gold fingers; ⑥ Network interface chip; ⑦ Memory module; ⑧ 25GBbE port; Figure 4 This is a simplified diagram of small nodes for individuals, families, and small businesses provided in an embodiment of the present invention.

[0027] ① 6-way PCIe 5.0 x4 cable; ② CPU / GPU and cache; ③ M.2 interface SSD (system disk); ④ 10GbE port; ⑤ Power supply gold fingers; ⑥ Network interface chip; ⑦ Memory module; Figure 5 This is a diagram showing the internal structure of a 2U 4-node 24-bay chassis provided in an embodiment of the present invention.

[0028] 1-Pallet; 2-Mother plate; 3-Small node card; 4-Fan wall; 5-Panel back plate. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The purpose of this invention is to provide a computing cluster architecture that can overcome bandwidth bottlenecks, significantly improve computing power utilization, and reduce costs.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example 1: This embodiment provides a high-bandwidth computing architecture and system based on storage hierarchy reconstruction.

[0033] like Figure 1 As shown, the GPU cluster system of this invention consists of thousands of standardized small nodes. The system adopts a hierarchical interconnect architecture: like Figure 2 As shown, in the rack installation layout, the 2U 24-bay chassis ① contains four in-memory compute nodes. Each node is equipped with six PCIe 5.0×4 SSDs and a 25GbE dual-port network interface. These nodes are interconnected with two 100GbE switches (i.e., top-mounted switches ④) on the rack via a 1-to-4 fiber optic cable. Switch connection port ③ is used for cable connections, and data transmission cable ⑤ is responsible for signal transmission. A 42U rack houses 16 2U 4-node servers. The two top-mounted switches ④ have 16 100GbE ports and 4 400GbE ports. These two top-mounted switches ④ are connected to two 400GbE backbone switches ② via 400GbE ports and fiber optic cables, enabling interconnection between racks and across data centers.

[0034] like Figure 3 As shown, each small node is a storage-computing unit. Specifically, ① is a 6-way PCIe 5.0 x4 cable for connecting the SSD; ② is the CPU / GPU and on-chip cache; ③ is the M.2 interface SSD (system drive); ④ is a 10GbE port (control plane); ⑤ is the power supply gold fingers; ⑥ is the network interface (southbridge) chip; ⑦ is the memory module; and ⑧ is a 25GbE port (data plane).

[0035] Based on the current PCIe 5.0 standard, its specific configuration is as follows: Processor module: includes CPU, GPU and on-chip cache②. Among them, the GPU has an FP16 computing power of no less than 1 TFLOPS and hardware supports double-packed Math.

[0036] Network interfaces: A dual-port 25GbE network interface is used as the data plane, and a dual-port 10GbE network interface ④ is used as the control plane.

[0037] Storage module: Configured with 6 hot-swappable PCIe 5.0 x4 SSDs installed in the front cage of the chassis, configured as RAID10 with 3-replica redundancy for local storage, with a single node read / write bandwidth of no less than 50GB / s.

[0038] The core of this architecture lies in the fact that the SSD serves as an on-chip cache directly connected to the processor. The working set is stored on the local SSD, and the processor (CPU+GPU) reads data directly from the SSD via a 6-way PCIe×4 cable from the CPU, including intermediate results. The calculation results are written directly back to the SSD, completely eliminating the frequent data copying between "memory and GPU memory" in traditional architectures.

[0039] like Figure 5 As shown, in the internal structure of the 2U 4-node 24-bay chassis, tray 1 supports the entire unit, motherboard 2 is fixed on tray 1, and four small node cards 3 are connected to the power supply slots of motherboard 2 via gold fingers. Fan wall 4 provides heat dissipation for the SSD and processor. The small node cards are connected via cables ( Figure 3 and Figure 4 ①) Connect the SSD to the backplate 5 of the drive cage.

[0040] Example 2: Using the computing cluster hardware described in Example 1, configured as a distributed network storage, it can pre-train large models with trillions or even tens of trillions of parameters. This example's cluster size is 10,000 small nodes, with a total of 60,000 SSDs. Each disk has a 4TB capacity, a raw capacity of 240PB, and is configured with 3 replicas for redundancy, resulting in an effective capacity of 80PB. Each small node has a theoretical GPU computing power of 1.2 TFLOPS (FP16), theoretically requiring 2.4TB of data to be loaded from the cache to the registers within one second, i.e., a theoretical cache bandwidth of 2.4TB / s. However, since the time to load data from the GPU cache to the registers is 30 times the computation time, the actual cache bandwidth is only 2.4 / 30 = 0.08TB / s. Because the computational intensity of matrix multiplication is O(N... 3 The standard for measuring computing power is the fusion of multiplication and addition operations, with a computational intensity of O(N). 2 Therefore, as long as the main memory can operate at a speed greater than or equal to 0.08... 3 / 2With approximately 0.023TB / s of bandwidth continuously providing data to the cache, the GPU's computing power utilization can reach 100%, with an effective computing power of 1.2 TFLOPS (FP16). Each small node's read bandwidth can reach 50GB / s = 0.05TB / s, more than twice the 0.023TB / s read bandwidth. With enough small nodes, unlimited computing power can be provided at 100% utilization. When the H100 cluster reaches a scale of 10,000 GPUs, its available bandwidth is the same as that of the small nodes, both at a unidirectional 50GB / s. Therefore, although the effective computing power of the small node GPU is only 71.6% of that of the H100, its main memory capacity is 100 times that of the H100 (8TB / 0.08TB), its procurement cost is 3.86% of the H100 ($22 million / $550 million), and its power consumption is 12% of the H100 (2MW / 18MW).

[0041] Example 3: In this embodiment, the cluster interconnection is designed based on the currently widely used PCIe 5.0 ×16 standard, as detailed below: like Figure 4 As shown, the small node is a streamlined architecture designed for individuals, families, and small businesses, including: ① 6-way PCIe 5.0 x4 cabling; ② CPU / GPU and cache; ③ M.2 interface SSD (system drive); ④ 10GbE port; ⑤ Power supply contacts; ⑥ Network interface chip; ⑦ Memory modules. The small node motherboard does not have a dual-port 25GbE network interface chip (i.e.,...) soldered on. Figure 3 (⑧) Only the 10GbE dual-port network interface chip (corresponding to ④ and ⑥) is soldered and connected to an external 10GbE switch. It is designed for individual, home, and small business customers. Users can start with a single small node host with 6 PCIe 5.0 x4 bandwidth, a computing power of no less than 1.2 TFLOPS (FP16), a raw capacity of 6TB, and a storage read bandwidth of no less than 50GB / s, and scale to large clusters consisting of tens of thousands of small nodes.

[0042] Example 4: The computing power cluster system of this invention (based on PCIe 5.0 × 16) is used to train a large model with trillions of parameters. The specific process is as follows: Data preparation: The training data and model parameters of the large model are directly stored in the SSD main memory of each small node. Through distributed network storage, a high-bandwidth, high-capacity, and 100% efficient computing power cluster is achieved.

[0043] Cluster Deployment: Deploy 10,000 small computing nodes to form a 10,000-calorie-level computing power cluster, through methods such as... Figure 1 and Figure 2The two-level network shown (including backbone switch ②, top-level switch ④, etc.) enables high-speed and high-reliability collaboration between nodes.

[0044] Training process: Each small node undertakes part of the model parameter training task. Data is directly read from the main memory composed of an all-flash distributed network system to the processor (CPU+GPU) cache for calculation. Intermediate calculation results are directly written to SSD for storage, without the need to set a checkpoint every 2-4 hours to back up intermediate results from main memory to SSD.

[0045] The foregoing has provided a detailed description of a high-bandwidth, high-utilization computing architecture, data processing method, and system based on storage hierarchy reconstruction provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0046] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0047] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A high-bandwidth computing architecture based on storage hierarchy reconstruction, characterized in that, include: Multiple standardized small nodes integrating storage and computing; Each of the aforementioned small nodes includes: The processor module includes a CPU with no fewer than 4 cores and a GPU that supports double speed and can pack two FP16 operations into one FP32 register for simultaneous execution. Main memory, using non-volatile SSD storage, is used to store the working set and computation results required for the computation; Cache, the high-speed cache built into the processor chip; The protocol adopts the latest version of the PCIe protocol; The architecture uses the SSD to construct a massive non-volatile storage system with a storage bandwidth consistent with the PCIe×16 bandwidth specified in the protocol as the main memory, and data is transferred directly between the caches without the need for data transfer between memory or video memory.

2. The high-bandwidth computing architecture based on storage hierarchy reconstruction according to claim 1, characterized in that, The storage module of the small node includes no less than four NVMe SSDs with the specified version of PCIe×4 bandwidth. Through striping technology, the storage bandwidth is equal to or close to the bandwidth of the specified version of PCIe×16, and data is not lost when two SSDs in a single node fail simultaneously.

3. The high-bandwidth computing architecture based on storage hierarchy reconstruction according to claim 1, characterized in that, The computing power of the GPU increases in tandem with the bandwidth of the PCIe version ×16 used in the interconnect interface; when using PCIe 5.0 ×16, the FP32 computing power of the GPU is not less than 0.075 TFLOPS. When using PCIe 6.0 ×16, the FP32 computing power of the GPU is no less than 0.15 TFLOPS.

4. The high-bandwidth computing architecture based on storage hierarchy reconstruction according to claim 1, characterized in that, It also includes a layered interconnect architecture: Inside the chassis, install one or more small nodes with a bidirectional storage bandwidth of not less than 50GB / s and a network bandwidth of not less than 50Gbps. The data plane is interconnected with two 100GbE leaf switches on the top of the rack via dual 25GbE ports and a 1-to-4 fiber optic cable. The control plane is interconnected with a 40GbE leaf switch on the top of the rack via dual 10GbE ports and a 1-to-4 copper cable. The racks and data centers are interconnected via a 400GbE backbone network. Two 100GbE leaf switches are interconnected with two 400GbE spine switches in a ratio of four 100GbE ports to one 400GbE port. Two 40GbE leaf switches are interconnected with a 400GbE spine switch in a ratio of ten 40GbE ports to one 400GbE port, forming a highly reliable dual-link Layer 2 leaf-spine network architecture with separate data plane and control plane.

5. The high-bandwidth computing architecture based on storage hierarchy reconstruction according to claim 4, characterized in that, When the small node adopts the PCIe 5.0 protocol, the high-speed Ethernet between the rack and the data center is 400GbE; when the small node adopts the PCIe 6.0 protocol, the high-speed Ethernet across the data center is 800GbE, and so on, so that the bandwidth of the large cluster is always consistent with the backbone network bandwidth.

6. A data processing method based on a high-bandwidth computing architecture based on storage hierarchy reconstruction as described in any one of claims 1 to 5, characterized in that, Includes the following steps: Data loading steps: The working set is stored in an SSD in local or distributed network storage, and during computation, it is read directly from the SSD in network or local storage into the processor's cache; Distributed computing steps: Multiple small nodes collaborate to compute through the network architecture described in claim 4. Each small node undertakes part of the computing task. Nodes inside the rack transmit data through a 100GbE top-mounted switch, and racks transmit data through a 400GbE backbone network switch. Result storage steps: Intermediate and final results generated during calculation are directly written to SSD main memory, with additional backups only performed when data needs to be permanently retained.

7. The data processing method according to claim 6, characterized in that, In the distributed computing step, there is no need to set checkpoints during the computing process to write intermediate results to non-volatile storage.

8. A high-bandwidth computing system based on storage hierarchy reconstruction, characterized in that, include: Multiple high-bandwidth computing architectures based on storage hierarchy reconfiguration as described in any one of claims 1 to 5; and an interconnection network connecting the multiple computing architectures; In this method, the computing power and storage capacity can be infinitely expanded by increasing the number of small nodes. The effective capacity of the main memory of a single node is 20 to 340,000 times that of the main memory of a single node in the existing computing power cluster, and the bandwidth is only limited by the bandwidth of the latest version of PCIe×16 that is currently widely used.

9. The computing system according to claim 8, characterized in that, The system can achieve 100% computing power utilization, and the system construction cost and operating power consumption are significantly reduced.

10. The computing system according to claim 8, characterized in that, When used for training large models, its training cycle is shortened by more than 30% compared to the training cycle using traditional GPU clusters.

Images