A data exchange network architecture for computing microsystems

Abstract

The present invention discloses a data exchange network architecture applied to a computing microsystem, comprising a plurality of distributedly designed hyper-converged nodes, each of which comprises an RDMA exchange module and a DDMA exchange node module, the RDMA exchange module and the DDMA exchange module are connected to each other, the DDMA exchange module is interconnected with a computing unit, a storage unit and an interface unit inside the hyper-converged node respectively, and the RDMA routing module is interconnected with a neighboring hyper-converged node outside the hyper-converged node, thereby realizing high-speed interconnection inside and outside nodes in a two-layer exchange network of DDMA and RDMA, and meeting the Internet network requirements of embedded distributed applications.

Description

Technical Field

[0001] The present invention relates to the field of high-bandwidth interconnected distributed technology, and in particular to a data exchange network architecture applied to a computing microsystem. Background Art

[0002] With the rapid development of information technology and the improvement of intelligence level, the demand for computing power, storage power and interface capabilities in embedded applications is growing rapidly. In application fields such as satellite communications, real-time remote sensing, and drone disaster relief, a few 1-2 information processing systems can no longer meet the needs of massive information processing. Therefore, high-end embedded applications inevitably adopt a distributed system architecture and integrate several or even dozens of high-performance information processing systems to meet the needs of computing, storage, and data analysis.

[0003] An important component that affects the performance of a distributed system is its data exchange network. For the Internet network of a distributed system, in addition to data transmission between adjacent nodes, the system also has high requirements for remote data access. The network technologies used in existing commercial distributed systems are mainly ultra-fast Ethernet and InfiniBand. Both types of Internet networks can achieve a data transmission rate of 10-100Gbps / channel, but generally only provide basic communication functions. Ultra-fast Ethernet devices are low-cost due to large-scale commercial use, but have poor performance in terms of network latency and CPU load. Although InfiniBand technology supports RDMA (Remote Direct Memory Access), its application scale is low, resulting in high costs, and it is usually only used in the most high-end distributed applications such as HPC.

[0004] Direct memory access technology (DMA) is one of the data transmission technologies of modern computers. Its main advantage is to avoid the frequent participation of the CPU in the data transmission process and improve the overall data transmission efficiency. RDMA is oriented to remote nodes and has the characteristics of low latency and low CPU load. It can be used to improve the Internet performance of distributed systems and effectively support the software functions of the application layer. In conventional distributed systems, distributed nodes usually use full-featured operating systems with relatively comprehensive software functions such as routing management, distributed file systems, and load balancing. However, embedded systems usually only have very simple real-time operating systems and do not have software functions such as routing management, distributed file systems, and load balancing. Therefore, ultra-high-speed Ethernet and InfiniBand cannot meet the design and application requirements of embedded distributed systems based on high-bandwidth interconnection in terms of software function support. It is necessary to build a new data exchange network and computing microsystem. Summary of the invention

[0005] The purpose of the present invention is to provide a data exchange network architecture applied to computing microsystems to overcome the problem that existing ultra-high-speed Ethernet and InfiniBand cannot functionally meet the application requirements of embedded distributed systems, realize high-speed interconnection inside and outside nodes, and support embedded distributed file systems and embedded task scheduling and migration systems at the software layer, thereby meeting the Internet requirements of embedded distributed applications.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A data exchange network architecture applied to a computing microsystem includes a plurality of distributedly designed hyper-converged nodes, each of which includes an RDMA exchange module and a DDMA exchange module, the RDMA exchange module and the DDMA exchange module are interconnected, the DDMA exchange module is interconnected with a computing unit, a storage unit and an interface unit inside the hyper-converged node respectively, and the RDMA exchange module is interconnected with a neighboring hyper-converged node outside the hyper-converged node.

[0008] Preferably, the RDMA switching module is interconnected with the RDMA switching module of a neighboring hyper-converged node.

[0009] Preferably, the hyper-converged nodes are interconnected using 2D-Mesh, 2D-Torus, or hypercube topology structures.

[0010] Preferably, the DDMA switching module inside the hyper-converged node adopts a 2-level fat tree topology.

[0011] Preferably, the leaf nodes in the 2-level fat tree topology are computing, storage and interface units, and the remaining nodes are task switches.

[0012] Preferably, the RDMA switching module is implemented by an RDMA switch, and the RDMA switch adopts a core switch and a two-stage switching method.

[0013] Preferably, the switching method adopted by the RDMA switch is that the first-level switching adopts multiple interconnected switches, each interconnected switch is further connected to multiple first-in-first-out queues for connecting to the input and output ports of other distributed nodes, and at the same time, each interconnected switch is connected to the core switch through its own main first-in-first-out queue, and the second-level switching adopts a core switch to access two internal first-in-first-out queues.

[0014] Preferably, the two internal first-in-first-out queues are used to connect the DDMA switch and the virtual network card respectively.

[0015] Preferably, the DDMA switching module is implemented by a DDMA switch, and the DDMA switch adopts a core switch and a two-stage switching method.

[0016] Preferably, the switching method adopted by the DDMA switch is that the first-level switching adopts multiple task switches, and each task switch is further connected to multiple first-in-first-out queues for connecting to the computing unit, storage unit and interface unit inside the hyper-converged node. At the same time, each task switch is connected to the core switch through its own main first-in-first-out queue. The second-level switching adopts a core switch to access an external first-in-first-out queue for connecting to the RDMA switch.

[0017] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a data exchange network architecture applied to a computing microsystem, including multiple distributed hyper-converged nodes, each hyper-converged node including an RDMA exchange module and a DDMA exchange module, the RDMA exchange module and the DDMA exchange module are connected to each other, the DDMA exchange module is interconnected with the computing unit, storage unit and interface unit inside the hyper-converged node respectively, the RDMA exchange module is interconnected with the neighboring hyper-converged node outside the hyper-converged node, realizing high-speed interconnection inside and outside the DDMA and RDMA two-layer exchange network nodes, and meeting the Internet network requirements of embedded distributed applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the DDMA+RDMA two-stage switching network of the present invention;

[0019] Figure 2 1 is a DDMA+RDMA network topology diagram of the present invention, wherein (a) is a DDMA inter-node topology diagram, and (b) is an RDMA intra-node topology diagram;

[0020] Figure 3 It is the RDMA switch design of the present invention;

[0021] Figure 4 It is the DDMA switch design of the present invention;

[0022] Figure 5 It is an embodiment of the data exchange network topology structure of the present invention;

[0023] Figure 6 It is an RDMA switch embodiment of the present invention;

[0024] Figure 7 This is an embodiment of the DDMA switch of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0026] like Figure 1-4The present invention provides a data exchange network architecture applied to a computing microsystem, including multiple distributed hyper-converged nodes, each hyper-converged node including an RDMA exchange module and a DDMA exchange module, the RDMA exchange module and the DDMA exchange module are connected to each other, the DDMA exchange module is interconnected with the computing unit, storage unit and interface unit inside the hyper-converged node respectively, and the RDMA exchange module is interconnected with the neighboring hyper-converged node outside the hyper-converged node.

[0027] In the present invention, RDMA is used to interconnect nodes; DDMA is used to interconnect devices within nodes (such as Figure 1 ). This design approach is compatible with different types of network topologies and different types of physical layer data transmission protocols (such as SRIO, PCIe, EMIB, UCIe, etc.).

[0028] In a distributed system, multiple buses may be used to interconnect nodes and units within nodes, and there are different designs for distributed network topologies. For an embedded data exchange network using DDMA+RDMA, the network topology design is related to multiple factors such as the number of nodes, the number of devices within the node, the DDMA switch design, the RDMA switch design, and the data transmission requirements of the network. With the support of three-dimensional integration technology, chip units can be stacked heterogeneously through high-density wiring. Therefore, in the present invention, the hyper-converged nodes support topologies such as 2D-Mesh, 2D-Torus, and Hypercube with high connectivity (such as Figure 2 ) are interconnected. The internal resources of the nodes are interconnected through a 2-level fat tree structure to ensure the bandwidth of data interaction. Among them, the leaf nodes are computing and storage units, and the remaining nodes are task switches.

[0029] RDMA switches are used to achieve peer-to-peer interaction between hyper-converged nodes. RDMA switches implement remote data forwarding between distributed nodes, so they mainly manage the external interconnection ports of distributed nodes and the high-speed ports that connect to the internal DDMA of the nodes. Since the number of external ports of RDMA switches is large, it is impossible to ensure that a high exchange rate can be achieved by using a single-stage switching structure in the chip. Therefore, the present invention adopts the technical route of core switches and two-stage switching for RDMA switches. The overall structure of the RDMA switch is as follows: Figure 3 As shown. The first-level switching uses multiple interconnected switches, each of which is further connected to multiple first-in-first-out queues (hereinafter collectively referred to as FIFOs) to connect to the input and output (hereinafter referred to as I / O) ports of other distributed nodes. At the same time, each interconnected switch is connected to the core switch through its own main FIFO. The second-level switching uses a core switch to access 2 internal FIFOs, which are used to connect to the DDMA switch and the virtual network card respectively.

[0030] The DDMA network switching structure is used to realize the peer-to-peer interaction between the internal resources of the hyper-converged node. The DDMA switch needs to be connected to the hardware port inside the node, which is very complex. First, the DDMA switch needs to be connected to the RDMA switch to open up the exchange of local data and remote data. Secondly, the DDMA switch also needs to connect to the computing CPU, the control CPU, the buffer in the coprocessing unit, and the memory of different capacities. Finally, the DDMA switch also needs to connect to a large number of high-speed and low-speed IO ports. The present invention also adopts the core switch and two-stage switching technology route for the DDMA switch.

[0031] The overall structure of a DDMA switch is as follows: Figure 4 As shown in the figure. To support different types of data exchange, the first-level exchange of the DDMA switch can be designed for different exchange task needs, and task switches of different types and rates can be designed. The structure of all task switches is the same as the interconnection switch in the RDMA switch, except that the bit width and transmission bandwidth of the interconnection bus and FIFO are different. The task switches are connected to the DDMA core switch through their respective main FIFOs. The second-level exchange uses a core switch to access 1 external FIFO for docking with the RDMA switch.

[0032] The present invention proposes to use DDMA and RDMA two-level switching networks at the MAC layer and the data link layer to realize data exchange. First, in DDMA, data is forwarded by a DDMA switch in the form of packets and transmitted between devices inside distributed nodes, thereby supporting concurrent and bidirectional data access between devices and improving the effective bandwidth of the data exchange network. Since the number of devices that can be mounted inside the node is limited, a relatively simplified switching strategy can be designed to achieve lower switching delay and higher switching bandwidth with fewer hardware resources inside the node. Secondly, in RDMA, a mature network routing strategy can be used to generate a forwarding information table in each RDAM switch, support fast data exchange between nodes, and support different types of remote data access. The two-level switching structure of the present invention can greatly reduce the data access delay between units inside the node on the one hand, and can greatly reduce the difficulty of implementing data access between remote nodes on the other hand.

[0033] Example:

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0035] This embodiment adopts heterogeneous distributed nodes and provides a topological structure example of an embedded data exchange network. Figure 5. Among them, the data exchange network includes a total of 9 distributed nodes, and data can be directly transmitted between adjacent directly connected nodes, and the number of data relay forwarding between non-adjacent nodes is not more than 1. The node has a total of 4 exits (1, 2, 3, 4), and the RDMA switch performs exit table query. The node includes hardware modules such as CPU (c1-c4), GPU, DSP, FPGA, and storage unit (m1-m4). The hardware modules are interconnected through the DDMA switch to support concurrent access to data. Both the RDMA switch and the DDMA switch are implemented using FPGA in the embodiment.

[0036] This embodiment uses a 160G core switch and two-stage switching technology for the RDMA switch. The first-stage switching uses three 40G interconnection switches, each of which is connected to 4 10G FIFOs for docking 4 ports. At the same time, each interconnection switch is connected to a 160G switch through a 160G FIFO. The second-stage switching uses a 160G switch, which is connected to 2 160G FIFOs and docked with the DDMA switch and the virtual network card respectively. The overall structure of the RDMA switch is as follows: Figure 6 shown.

[0037] This embodiment also uses the 160G core switch and two-stage switching technology route for the DDMA switch. To support different types of data exchange, the first-stage switching of the DDMA switch uses three types of task switches, including: an 80G computing switch, a 40G high-speed IO switch, and a 10G low-speed IO switch. The internal structure of all task switches is the same as the interconnection switch in the RDMA switch, except that the bit width and bandwidth of the interconnection bus and FIFO are different. Among them, the computing switch is connected to FIFOs with capacities of 40G, 20G, and 20G, respectively, which are used to connect to SRIOx4, PCIex4, and co-processing modules respectively; the high-speed IO switch is connected to 4 10G FIFOs, which are used to connect to 4 high-speed IO ports respectively; the low-speed IO switch is connected to 4 1G FIFOs, which are used to connect to 4 low-speed IO ports respectively. At the same time, the computing switch, high-speed IO switch, and low-speed IO switch are connected to the 160G switch through 160GFIFO. The second-stage switching uses a 160G switch to access a 160G FIFO and connect to the RDMA switch. The overall structure of the DDMA switch is as follows Figure 7 shown.

[0038] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of the specification, ordinary technicians in this field can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A data exchange network architecture for a computing microsystem, characterized in that: It includes multiple distributed hyper-converged nodes, each of which includes an RDMA switching module and a DDMA switching module, which are connected to each other, the DDMA switching module is interconnected with a computing unit, a storage unit and an interface unit inside the hyper-converged node respectively, and the RDMA switching module is interconnected with a neighboring hyper-converged node outside the hyper-converged node.

2. The data exchange network architecture applied to a computing microsystem according to claim 1, characterized in that: The RDMA switching module is interconnected with the RDMA switching module of the neighboring hyper-converged node.

3. The data exchange network architecture applied to a computing microsystem according to claim 1, characterized in that: The hyper-converged nodes are interconnected using 2D-Mesh, 2D-Torus, and hypercube topology structures.

4. The data exchange network architecture applied to a computing microsystem according to claim 1, characterized in that: The DDMA switching module inside the hyper-converged node adopts a two-level fat tree topology.

5. The data exchange network architecture applied to a computing microsystem according to claim 4, characterized in that: The leaf nodes in the two-level fat tree topology are computing, storage and interface units, and the remaining nodes are task switches.

6. The data exchange network architecture applied to a computing microsystem according to claim 1, characterized in that: The RDMA switching module is implemented by an RDMA switch, and the RDMA switch adopts a core switch and a two-stage switching method.

7. The data exchange network architecture applied to a computing microsystem according to claim 6, characterized in that: The switching method adopted by the RDMA switch is that the first-level switching uses multiple interconnected switches, and each interconnected switch is further connected to multiple first-in-first-out queues for connecting to the input and output ports of other distributed nodes. At the same time, each interconnected switch is connected to the core switch through its own main first-in-first-out queue. The second-level switching uses a core switch to access two internal first-in-first-out queues.

8. The data exchange network architecture applied to a computing microsystem according to claim 7, characterized in that: The two internal first-in-first-out queues are used to connect the DDMA switch and the virtual network card respectively.

9. The data exchange network architecture applied to a computing microsystem according to claim 1, characterized in that: The DDMA switching module is implemented by a DDMA switch, and the DDMA switch adopts a core switch and a two-stage switching method.

10. The data exchange network architecture applied to a computing microsystem according to claim 9, characterized in that: The switching method adopted by the DDMA switch is that the first-level switching uses multiple task switches, and each task switch is further connected to multiple first-in-first-out queues for connecting to the computing unit, storage unit and interface unit inside the hyper-converged node. At the same time, each task switch is connected to the core switch through its own main first-in-first-out queue. The second-level switching uses a core switch to access an external first-in-first-out queue for connecting to the RDMA switch.

Images