Method, system, device and storage medium for interconnecting multi-chip cluster servers

Abstract

This invention provides a method, system, device, and storage medium for interconnecting multi-chip cluster servers. The method includes: setting a custom-designed chip comprising multiple PCIe terminal devices and a first PCIe bridge device, and setting a custom-designed switching chip comprising a PCIe ROOT port function module and a second PCIe bridge device; setting the custom-designed chip in each server cluster, establishing a connection between each SOC server in the server cluster and the custom-designed chip through the multiple PCIe terminal devices, and establishing a connection between the second PCIe bridge device and each first PCIe bridge device; in response to data interaction within the cluster, establishing a first routing channel between the two servers through the first PCIe bridge device, and performing data interaction through the first routing channel; and in response to data interaction between clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

Description

Technical Field

[0001] This invention relates to the field of chip design, and more specifically, to a method, system, device, and storage medium for interconnecting multi-chip cluster servers. Background Technology

[0002] With the rapid development and increasing scale of cloud computing in recent years, the application of multi-processor servers has become increasingly widespread. These servers offer high stability, security, and computing efficiency, and have been widely adopted in almost all aspects of production and daily life. As server applications continue to evolve, the demand for high-end servers has entered a crucial stage. Complex architectures are needed to support high-performance server systems, enabling them to achieve high security, high availability, and high reliability.

[0003] Current implementations of shared memory based on cluster interconnects of ordinary processors, such as Figure 1 As shown, processor clusters are typically formed by interconnecting multiple levels of network switches via standard Ethernet, and various system tasks are executed through task scheduling in system software. Specifically, in the interconnection of processors in a mobile phone system chip, since its SOC (System-on-Chip) generally does not have a standard Ethernet interface, such as... Figure 2 As shown, the common practice is to connect an external network conversion chip via PCIe (Peripheral Component Interconnect Express) or USB interface to convert the interface to a standard Ethernet interface, and then interconnect them through a network switch to form a cluster system with the same architecture as ordinary processors interconnected via Ethernet interface.

[0004] For general-purpose servers using server-grade processor chips, there are enough network interfaces to connect to Ethernet switches of various levels, thus enabling multi-machine clusters. However, for clusters implemented with low-cost mobile phone SOC chips, the lack of standard network interfaces limits their connectivity. Figure 2It is known that external network interface adapter chips are required to achieve multi-processor network clustering. Since the purpose of using low-cost mobile SoC chips for clustering is to reduce the overall cost of the processor cluster, the additional network interface adapter chips for each mobile SoC chip significantly increase the overall system cost, thus defeating the initial system design goal of reducing the overall processor cluster cost. Furthermore, interface conversion requires multi-level forwarding across multiple different interfaces, and the efficiency of interconnection between mobile SoC chips is limited by each interface, significantly reducing communication and data exchange efficiency. Therefore, using traditional mobile SoC chip cluster interconnection solutions will result in a significant performance reduction, and the cost of the cluster system cannot be significantly reduced, significantly impacting the cost-effectiveness and market competitiveness of the entire system solution. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, system, computer device, and computer-readable storage medium for interconnecting multi-chip cluster servers. This invention simplifies the interconnection of SOC processors, thereby significantly reducing the number of external expansion chips and achieving a substantial reduction in product costs. This invention creatively adopts a new method to realize the clustering of SOC processors, providing new ideas and product approaches for the design and implementation of micro-cluster servers. This invention greatly simplifies the complexity of SOC processor cluster interconnection, reduces design resource overhead, thereby effectively reducing production and testing costs. The simpler functional implementation also effectively reduces the implementation complexity during the development and testing process.

[0006] Based on the above objectives, one aspect of the present invention provides a method for interconnecting multi-chip cluster servers, comprising the following steps: setting a custom-designed chip including multiple PCIe terminal devices and a first PCIe bridge device, and setting a custom-designed switching chip including a PCIe ROOT port function module and a second PCIe bridge device; setting a custom-designed chip in each server cluster, establishing a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establishing a connection between the second PCIe bridge device and each of the first PCIe bridge devices; in response to data interaction within the cluster, establishing a first routing channel between the two servers through the first PCIe bridge device, and performing data interaction through the first routing channel; and in response to data interaction between clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

[0007] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the originating server writing data to the destination server, initiating a read data request based on a specified memory address of the originating server, and initiating a DMA write memory request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to the PCIe terminal device of the destination server according to configuration information; the PCIe terminal device of the destination server initiates a DMA write request to the destination server and receives return information from the destination server; and the first PCIe bridge device sends the write memory address in the return information to the originating server to establish a first routing channel from the address space of the originating server to the address space of the destination server.

[0008] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the starting server reading data from the destination server, configuring a DMA write memory instruction for the PCIe terminal device of the starting server and allocating a write memory address space, data size, and address information of the destination server; and the first PCIe bridge device routing the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, wherein the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0009] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster writing data to the destination cluster, initiating a read data request based on a specified memory address of the starting server of the starting cluster, and initiating a DMA write memory request to the first PCIe bridge device of the starting server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the destination cluster via the second PCIe bridge device, and the first PCIe bridge device of the destination cluster routes the DMA write memory request to the PCIe terminal device of the destination server based on memory space configuration information; and receiving the return information from the destination server via the PCIe terminal device of the destination server, and routing the write memory address in the return information to the PCIe terminal device of the starting server via the second PCIe bridge device, the first PCIe bridge device of the starting cluster, and the first PCIe bridge device of the destination cluster.

[0010] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster reading data from the destination cluster, the starting cluster configures a DMA write memory instruction for the connected PCIe terminal device and allocates a write memory address space, data size, and address information of the destination server of the destination cluster; the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster via the second PCIe bridge device according to the configuration information; the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0011] In some implementations, the method further includes: forming a basic PCIe system with each PCIe terminal device and a corresponding server, and independently initializing each basic PCIe system, allocating a corresponding ID and address space, and binding it to a corresponding custom-designed chip for use.

[0012] In some implementations, the method further includes: forming a tree-like PCIe system with each PCIe terminal device, a first PCIe bridge device, a second PCIe bridge device, and a root module with a ROOT port, and allocating independent and different buses, device IDs, and address spaces to each level of bridge devices and all PCIe terminal devices in each tree-like PCIe system.

[0013] In another aspect, this invention provides a system for interconnecting multi-chip cluster servers, comprising: a setting module configured to set a custom-designed chip including multiple PCIe terminal devices and a first PCIe bridge device, and to set a custom switching chip including a PCIe ROOT port function module and a second PCIe bridge device; a connection module configured to set the custom-designed chip in each server cluster, establish a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establish a connection between the second PCIe bridge device and each of the first PCIe bridge devices; a first interaction module configured to, in response to data interaction within the cluster, establish a first routing channel between two servers through the first PCIe bridge device, and perform data interaction through the first routing channel; and a second interaction module configured to, in response to data interaction between clusters, establish a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and perform data interaction through the second routing channel.

[0014] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of the method described above.

[0015] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0016] The present invention has the following beneficial technical effects:

[0017] 1. Through a custom design of a multi-level PCIe bridge, data exchange between various mobile phone SOC processor chips is realized, replacing the additional conversion chip on each interface. Data exchange within the cluster nodes no longer requires each mobile phone SOC processor chip to be connected to the network switch as a single node, thereby significantly reducing the implementation cost of the cluster.

[0018] 2. Through the interconnection of the PCIe interface, multiple mobile SoC processors can directly perform P2P memory sharing and data exchange via PCIe EP devices, reducing the significant latency caused by multiple interface forwardings and improving overall system performance.

[0019] 3. Enables fast data exchange between multiple PCIe root nodes, providing a new fast data exchange solution for processor interconnection with PCIe root interfaces;

[0020] 4. Easily expand the number of SOC processors in a cluster. Processors in different nodes can achieve fast data exchange across nodes through a multi-PCIe tree system composed of terminal EP devices. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the architecture for cluster interconnection of ordinary processors in the prior art;

[0023] Figure 2 This is a schematic diagram of the architecture for interconnecting a network switch and a System-on-a-Chip (SoC) in the existing technology.

[0024] Figure 3A schematic diagram illustrating an embodiment of the method for interconnecting multi-chip cluster servers provided by the present invention;

[0025] Figure 4 This is a schematic diagram of the architecture for interconnecting multi-chip cluster servers provided by the present invention;

[0026] Figure 5 A schematic diagram illustrating an embodiment of the multi-chip cluster server interconnection system provided by the present invention;

[0027] Figure 6 A schematic diagram of the hardware structure of a computer device for interconnecting multi-chip cluster servers provided by the present invention;

[0028] Figure 7 This is a schematic diagram of an embodiment of the computer storage medium for interconnecting multi-chip cluster servers provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0030] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0031] In a first aspect, an embodiment of a method for interconnecting multi-chip cluster servers is proposed. Figure 3 The diagram illustrates an embodiment of the method for interconnecting multi-chip cluster servers provided by the present invention. Figure 3 As shown, the embodiments of the present invention include the following steps:

[0032] S1. Set up a custom-designed chip including multiple PCIe terminal devices and a first PCIe bridge device, and set up a custom switching chip including a PCIe ROOT port function module and a second PCIe bridge device.

[0033] S2. Set up a custom-designed chip in each server cluster, establish a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establish a connection between the second PCIe bridge device and each of the first PCIe bridge devices.

[0034] S3. In response to data interaction within the cluster, a first routing channel is established between the two servers through the first PCIe bridge device, and data interaction is performed through the first routing channel; and

[0035] S4. In response to data interaction between clusters, a second routing channel is established between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and data interaction is performed through the second routing channel.

[0036] This invention considers using ARM-based mobile SoC chips to achieve task distribution and data exchange among multiple processors. This greatly expands the application scenarios of ARM-based mobile SoC chips and enables the miniaturization of ARM-based mobile SoC chip-based server clusters. It achieves the processing functions of a general-purpose ARM server within a small physical mechanical size. Due to the large production volume and market share of ARM-based mobile SoC chips, processors implemented in this way have a natural cost advantage under large-scale production. Furthermore, ARM-based mobile SoC chip-based server clusters have a natural advantage in compatibility and adaptability for mobile applications running on the server side.

[0037] This invention provides a novel method for rapid data interaction of multiple SoC chips in a secondary cluster interconnection on a single board. This method facilitates rapid data interaction between multiple SoC chips directly using the interface of an ARM processor-based mobile SoC chip, enabling rapid task dispatch across multiple SoC chips and ensuring stable application operation. While maintaining the SoC chip interface unchanged, it is compatible with both mobile and server-side use, expanding the application scenarios of ARM processor-based mobile SoC chips. Furthermore, it matches the iteration speed of mobile SoC chips, enabling rapid mass production and commercialization of ARM processor-based mobile SoC chip server cluster devices.

[0038] Figure 4 This is a schematic diagram of the multi-chip cluster server interconnection architecture provided by the present invention, as shown below. Figure 4 As shown, this invention comprises m cluster servers, each cluster containing n SOC server systems. The SOC servers are connected to the custom-designed cluster chip of this invention via PCIe interfaces. The interface on the chip connecting to each SOC server is presented as an EP (terminal) device, and the ROOT port of the SOC server is connected to it via a PCIe interface. Each cluster contains one custom-designed chip of this invention connected to each SOC server within the cluster via a PCIe interface. A total of m custom-designed chips of this invention exist in the system, each connected to one of the m SOC servers in the cluster via a PCIe interface.

[0039] Each cluster contains a custom-designed chip from this invention that includes a total of PCIe Bridge devices. Figure 4The bridge1 component contains n PCIe EP (End Point) devices. Each EP device has a PCIe interface in two directions: one direction, as described above, connects to the ROOT interface of the corresponding SOC server, and the other direction connects to the downstream port of the bridge device in the chip. The bridge device in the chip contains a total of n downstream ports and one upstream port.

[0040] M server clusters are interconnected through the customized switching chip described in this invention. Figure 4 The bottommost diagram shows the custom switching chip, which includes a PCIe ROOT port module and a PCIe Bridge device. Figure 4 Bridge0 (in the context of the cluster) has m downstream ports, which are connected to the upstream ports of the m custom-chip-based Bridge1 bridge devices in the cluster. The upstream ports of Bridge0 are connected to the ROOT module.

[0041] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the originating server writing data to the destination server, initiating a read data request based on a specified memory address of the originating server, and initiating a DMA write memory request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to the PCIe terminal device of the destination server according to configuration information; the PCIe terminal device of the destination server initiates a DMA write request to the destination server and receives return information from the destination server; and the first PCIe bridge device sends the write memory address in the return information to the originating server to establish a first routing channel from the address space of the originating server to the address space of the destination server.

[0042] The following example illustrates the process of writing data from SOC0's memory to SOC1's memory. SOC0 configures a DMA read memory command to the corresponding EP0 via its ROOT0 port. EP0 initiates its built-in DMA, sending a read data request based on the specified memory address. Simultaneously, EP0 sends a DMA write memory request to Bridge1. Bridge1 routes the DMA write memory request to EP1 based on its configuration information. EP1 then sends a DMA write request to SOC1. SOC1 responds to this request, allocates the corresponding data storage address space, and returns this address space information to EP1's DMA. Upon receiving this information, EP1 sends the corresponding write memory address information to EP0 via Bridge1, establishing a routing channel from SOC0's address space to SOC1's address space. Then, EP0 reads data from SOC0's corresponding address space and writes it to SOC1's corresponding address space. Once the data writing is complete, EP0 and EP1 respectively send interrupts to SOC0 and SOC1's systems, informing them that the read and write operations are complete.

[0043] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the starting server reading data from the destination server, configuring a DMA write memory instruction for the PCIe terminal device of the starting server and allocating a write memory address space, data size, and address information of the destination server; and the first PCIe bridge device routing the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, wherein the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0044] The following example illustrates how SOC0 reads data from SOC1's memory into SOC0's memory. SOC0 configures a DMA write memory command to the corresponding EP0 via the ROOT0 port, allocating the write memory address space, data size, and the address information of SOC1 to be read. EP0 initiates a DMA read memory request to Bridge1 below. Bridge1 routes the DMA read memory request to EP1 based on the address configuration information. EP1 then initiates a DMA read request to SOC1. SOC1 responds to the request, returning the read memory data to EP1. EP1 sends the read data to EP0, and EP0's DMA initiates a write request, writing the data to the corresponding memory space of SOC0. After all data is written, EP0 and EP1 respectively send interrupts to the systems of SOC0 and SOC1, notifying them that the write and read operations are complete.

[0045] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster writing data to the destination cluster, initiating a read data request based on a specified memory address of the starting server of the starting cluster, and initiating a DMA write memory request to the first PCIe bridge device of the starting server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the destination cluster via the second PCIe bridge device, and the first PCIe bridge device of the destination cluster routes the DMA write memory request to the PCIe terminal device of the destination server based on memory space configuration information; and receiving the return information from the destination server via the PCIe terminal device of the destination server, and routing the write memory address in the return information to the PCIe terminal device of the starting server via the second PCIe bridge device, the first PCIe bridge device of the starting cluster, and the first PCIe bridge device of the destination cluster.

[0046] The following example illustrates the process of writing data from the memory of SOC0 in cluster 1 to the memory of SOC1 in cluster 2. SOC0 in cluster 1 configures a DMA read memory command to the corresponding EP0 via the ROOT0 port. EP0 initiates its built-in DMA, sending a read data request based on the specified memory address of SOC0 in cluster 1. Simultaneously, EP0 sends a DMA write memory request to Bridge1. Bridge1 forwards the request to Bridge0 based on the memory space configuration information. Bridge0 then routes the DMA write memory request to Bridge1 in cluster 2 based on the memory space configuration information. Bridge1 then routes the DMA write memory request to EP1 downstream based on the memory space configuration information. EP1 sends a DMA write request to the connected SOC1 in cluster 2. SOC1 in cluster 2 responds to the request, allocates the corresponding data storage address space, and returns this address space information to EP1's DMA. Upon receiving this information, EP1 routes the corresponding write memory address information to EP0 in cluster 0 via Bridge1 and Bridge0, establishing a routing channel from the address space of SOC0 in cluster 1 to the address space of SOC1 in cluster 2. Then, EP0 reads data from the corresponding address space of SOC0 in cluster 1. After receiving the data, it writes the data into the corresponding address space of SOC1 in cluster 2. After all the data is written, it sends interrupts to the systems of SOC0 in cluster 1 and SOC1 in cluster 2 respectively to inform them that reading and writing data are complete.

[0047] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster reading data from the destination cluster, the starting cluster configures a DMA write memory instruction for the connected PCIe terminal device and allocates a write memory address space, data size, and address information of the destination server of the destination cluster; the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster via the second PCIe bridge device according to the configuration information; the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0048] The following example illustrates how SOC0 of cluster 1 reads data from the memory of SOC1 of cluster 2 into the memory of SOC0 of cluster 1. SOC0 of cluster 1 configures a DMA write memory command through its ROOT0 port to the corresponding EP0, allocating the write memory address space, data size, and the address information of SOC1 of cluster 2 to be read. EP0 initiates a DMA read memory request to Bridge1. Bridge1, based on the address configuration information, sends the DMA read memory request upstream to Bridge0. Bridge0, based on the address configuration information, routes the DMA read memory request to Bridge1 of cluster 2. Bridge1 of cluster 2, based on the address configuration information, routes the DMA read memory request to EP1 of cluster 2. EP1 initiates a DMA read request to SOC1 of cluster 2. SOC1 of cluster 2 responds to the request, returning the read memory data to EP1. EP1 sends the read data to Bridge1 of cluster 2. Bridge1 forwards the data to Bridge0. Bridge0 forwards the data to EP0 of cluster 1. After receiving the data, EP0 initiates a write request through its built-in DMA, writing the data into the corresponding memory space of SOC0 of cluster 1. After all data has been written, EP0 of cluster 1 and EP1 of cluster 2 send interrupts to SOC0 of cluster 1 and SOC1 of cluster 2 respectively to notify that the writing and reading of data are complete.

[0049] During system initialization, Figure 4Each EP (Extended Processing Unit) is presented in two PCIe systems through its two PCIe interfaces. One system consists of the EP and the ROOT port of the SOC server above, forming a ROOT-EP basic PCIe system to realize data interaction and communication between the SOC server and the EP port. The other system consists of the EP connected to the switch chip's bridge0 through bridge1 below, and then connected to the root module with the ROOT port below through bridge0. This forms a full PCIe tree system structure containing ROOT-bridge0-brgidg1*m-EP*m*n.

[0050] In some implementations, the method further includes: forming a basic PCIe system with each PCIe terminal device and a corresponding server, and independently initializing each basic PCIe system, allocating a corresponding ID and address space, and binding it to a corresponding custom-designed chip for use. Figure 4 There are a total of m*n ROOT-EP PCIe systems above. Each PCIe system is initialized independently, assigned its own ID number and address space, and bound to the chip number through custom design.

[0051] In some implementations, the method further includes: forming a tree-like PCIe system with each PCIe terminal device, a first PCIe bridge device, a second PCIe bridge device, and a root module with a ROOT port, and allocating independent and different buses, device IDs, and address spaces to each level of bridge devices and all PCIe terminal devices in each tree-like PCIe system. Figure 4 Below is a tree-structured PCIe system. During initialization, the system traversal is initiated by the ROOT port at the bottom to allocate independent and different bus, device ID number, address space and other configuration information to each level of bridge device and all EP terminal devices.

[0052] This invention simplifies the interconnection of SOC processors, thereby significantly reducing the number of external expansion chips and achieving a substantial reduction in product costs. This invention creatively employs a new method to achieve SOC processor clustering, providing new ideas and product approaches for the design and implementation of micro-cluster servers. This invention greatly simplifies the complexity of SOC processor cluster interconnection, reduces design resource overhead, and thus effectively reduces production and testing costs. The simpler functional implementation also effectively reduces the complexity of implementation during the development and testing process.

[0053] It should be particularly noted that the steps in each embodiment of the above-described method for interconnecting multi-chip cluster servers can be interchanged, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the method for interconnecting multi-chip cluster servers should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the embodiments.

[0054] Based on the above objectives, a second aspect of this invention proposes a system for interconnecting multi-chip cluster servers. For example... Figure 5 As shown, system 200 includes the following modules: a setting module, configured to set a custom-designed chip including multiple PCIe terminal devices and a first PCIe bridge device, and to set a custom-designed switching chip including a PCIe ROOT port function module and a second PCIe bridge device; a connection module, configured to set a custom-designed chip in each server cluster, establish a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establish a connection between the second PCIe bridge device and each of the first PCIe bridge devices; a first interaction module, configured to establish a first routing channel between two servers through the first PCIe bridge device and perform data interaction in response to data interaction within the cluster; and a second interaction module, configured to establish a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device and perform data interaction in response to data interaction between clusters.

[0055] In some implementations, the first interaction module is configured to: in response to the originating server writing data to the destination server, initiate a read data request based on a specified memory address of the originating server, and initiate a DMA write memory request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to the PCIe terminal device of the destination server according to configuration information, the PCIe terminal device of the destination server initiates a DMA write request to the destination server, and receives return information from the destination server; and send the write memory address in the return information to the originating server through the first PCIe bridge device to establish a first routing channel from the address space of the originating server to the address space of the destination server.

[0056] In some implementations, the first interaction module is configured to: in response to the starting server reading data from the destination server, configure a DMA write memory instruction for the PCIe terminal device of the starting server and allocate a write memory address space, data size and address information of the destination server; and the first PCIe bridge device routes the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0057] In some implementations, the second interaction module is configured to: in response to the starting cluster writing data to the destination cluster, initiate a read data request based on a specified memory address of the starting server of the starting cluster, and initiate a DMA write memory request to the first PCIe bridge device of the starting server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the destination cluster through the second PCIe bridge device, and the first PCIe bridge device of the destination cluster routes the DMA write memory request to the PCIe terminal device of the destination server based on memory space configuration information; and receive the return information from the destination server through the PCIe terminal device of the destination server, and route the write memory address in the return information to the PCIe terminal device of the starting server through the second PCIe bridge device, the first PCIe bridge device of the starting cluster, and the first PCIe bridge device of the destination cluster.

[0058] In some implementations, the second interaction module is configured to: in response to the starting cluster reading data from the destination cluster, configure a DMA write memory instruction for the connected PCIe terminal device and allocate a write memory address space and data size, as well as the address information of the destination server of the destination cluster; the first PCIe bridge device of the starting server routes the DMA read memory request through the second PCIe bridge device to the PCIe terminal device of the destination server of the destination cluster according to the configuration information; the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0059] In some implementations, the system also includes an initialization module configured to: combine each PCIe terminal device with a corresponding server to form a basic PCIe system, and to independently initialize each basic PCIe system, allocate a corresponding ID and address space, and bind it to a corresponding custom-designed chip.

[0060] In some implementations, the system also includes an allocation module configured to: form a tree-like PCIe system with each PCIe terminal device, a first PCIe bridge device, a second PCIe bridge device, and a root module with a ROOT port, and allocate independent and different buses, device IDs, and address spaces to each level of bridge devices and all PCIe terminal devices in each tree-like PCIe system.

[0061] Based on the above objectives, a third aspect of the present invention provides a computer device comprising: at least one processor; and a memory storing computer instructions executable by the processor to perform the following steps: S1, setting a custom-designed chip including multiple PCIe terminal devices and a first PCIe bridge device, and setting a custom-designed switching chip including a PCIe ROOT port function module and a second PCIe bridge device; S2, setting a custom-designed chip in each server cluster, establishing a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establishing a connection between the second PCIe bridge device and each of the first PCIe bridge devices; S3, in response to data interaction within the cluster, establishing a first routing channel between two servers through the first PCIe bridge device, and performing data interaction through the first routing channel; and S4, in response to data interaction between clusters, establishing a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

[0062] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the originating server writing data to the destination server, initiating a read data request based on a specified memory address of the originating server, and initiating a DMA write memory request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to the PCIe terminal device of the destination server according to configuration information; the PCIe terminal device of the destination server initiates a DMA write request to the destination server and receives return information from the destination server; and the first PCIe bridge device sends the write memory address in the return information to the originating server to establish a first routing channel from the address space of the originating server to the address space of the destination server.

[0063] In some implementations, establishing a first routing channel between two servers via the first PCIe bridge device includes: in response to the starting server reading data from the destination server, configuring a DMA write memory instruction for the PCIe terminal device of the starting server and allocating a write memory address space, data size, and address information of the destination server; and the first PCIe bridge device routing the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, wherein the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0064] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster writing data to the destination cluster, initiating a read data request based on a specified memory address of the starting server of the starting cluster, and initiating a DMA write memory request to the first PCIe bridge device of the starting server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the destination cluster via the second PCIe bridge device, and the first PCIe bridge device of the destination cluster routes the DMA write memory request to the PCIe terminal device of the destination server based on memory space configuration information; and receiving the return information from the destination server via the PCIe terminal device of the destination server, and routing the write memory address in the return information to the PCIe terminal device of the starting server via the second PCIe bridge device, the first PCIe bridge device of the starting cluster, and the first PCIe bridge device of the destination cluster.

[0065] In some implementations, establishing a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: in response to the starting cluster reading data from the destination cluster, the starting cluster configures a DMA write memory instruction for the connected PCIe terminal device and allocates a write memory address space, data size, and address information of the destination server of the destination cluster; the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster via the second PCIe bridge device according to the configuration information; the PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

[0066] In some implementations, the steps further include: forming a basic PCIe system with each PCIe terminal device and a corresponding server, and independently initializing each basic PCIe system, allocating a corresponding ID and address space, and binding it to a corresponding custom-designed chip.

[0067] In some implementations, the steps further include: forming a tree-like PCIe system with each PCIe terminal device, a first PCIe bridge device, a second PCIe bridge device, and a root module with a ROOT port, and allocating independent and different buses, device IDs, and address spaces to each level of bridge devices and all PCIe terminal devices in each tree-like PCIe system.

[0068] like Figure 6 The diagram shown is a hardware structure schematic of an embodiment of the computer device for interconnecting multi-chip cluster servers provided by the present invention.

[0069] For example Figure 6Taking the device shown as an example, the device includes a processor 301 and a memory 302.

[0070] Processor 301 and memory 302 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0071] The memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-chip cluster server interconnection method in this embodiment. The processor 301 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 302, thereby realizing the multi-chip cluster server interconnection method.

[0072] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of a multi-chip cluster server interconnection method, etc. Furthermore, memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] The computer instructions 303 corresponding to the method of interconnecting one or more multi-chip cluster servers are stored in the memory 302. When executed by the processor 301, the method of interconnecting multi-chip cluster servers in any of the above method embodiments is executed.

[0074] Any embodiment of the computer device that performs the above-described method for interconnecting multi-chip cluster servers can achieve the same or similar effects as any of the aforementioned method embodiments.

[0075] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a method for interconnecting a multi-chip cluster server.

[0076] like Figure 7 The diagram shown is a schematic representation of an embodiment of the computer storage medium for interconnecting multi-chip cluster servers provided by the present invention. Figure 7Taking the computer storage medium shown as an example, the computer-readable storage medium 401 stores a computer program 402 that, when executed by a processor, performs the above method.

[0077] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for the method of interconnecting multi-chip cluster servers can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above embodiments of the computer program can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0078] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0079] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0080] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0081] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0082] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for interconnecting multi-chip cluster servers, characterized in that, Includes the following steps: The system includes custom-designed chips for multiple PCIe terminal devices and a first PCIe bridge device, as well as custom-designed switching chips for the PCIe ROOT port function module and the second PCIe bridge device. A custom-designed chip is set in each server cluster. A connection is established between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices. The second PCIe bridge device is also connected to each of the first PCIe bridge devices. In response to data interaction within the cluster, a first routing channel is established between the two servers through the first PCIe bridge device, and data interaction is performed through the first routing channel; as well as In response to data interaction between clusters, a second routing channel is established between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and data interaction is carried out through the second routing channel.

2. The method according to claim 1, characterized in that, The establishment of the first routing channel between the two servers through the first PCIe bridge device includes: In response to the originating server writing data to the destination server, a read data request is initiated based on the memory address specified by the originating server, and a DMA write memory request is initiated to the first PCIe bridge device. The first PCIe bridge device routes the DMA write memory request to the PCIe terminal device of the destination server according to the configuration information. The PCIe terminal device of the destination server initiates a DMA write request to the destination server and receives the return information from the destination server. The first PCIe bridge device sends the write memory address in the returned information to the originating server to establish a first routing channel from the originating server address space to the destination server address space.

3. The method according to claim 1, characterized in that, The establishment of the first routing channel between the two servers through the first PCIe bridge device includes: In response to the originating server reading data from the destination server, the system configures a DMA write memory instruction for the PCIe terminal device of the originating server and allocates the write memory address space, data size, and address information of the destination server; and The first PCIe bridge device routes the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information. The PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

4. The method according to claim 1, characterized in that, The establishment of a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: In response to the starting cluster writing data to the destination cluster, a read data request is initiated based on the specified memory address of the starting server of the starting cluster, and a DMA write memory request is initiated to the first PCIe bridge device of the starting server. The first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the destination cluster through the second PCIe bridge device. The first PCIe bridge device of the destination cluster then routes the DMA write memory request to the PCIe terminal device of the destination server according to the memory space configuration information. The PCIe terminal device of the destination server receives the return information from the destination server, and routes the write memory address in the return information to the PCIe terminal device of the originating server through the second PCIe bridge device, the first PCIe bridge device of the originating cluster, and the first PCIe bridge device of the destination cluster.

5. The method according to claim 1, characterized in that, The establishment of a second routing channel between the two clusters via the first PCIe bridge device and the second PCIe bridge device includes: In response to the starting cluster reading data from the destination cluster, the starting cluster configures the DMA write memory command for the connected PCIe terminal device and allocates the write memory address space, data size, and address information of the destination server of the destination cluster. The first PCIe bridge device of the starting server routes the DMA read memory request through the second PCIe bridge device to the PCIe terminal device of the destination server in the destination cluster according to the configuration information. The PCIe terminal device of the destination server initiates a DMA read request to the destination server and receives the return data from the destination server.

6. The method according to claim 1, characterized in that, The method also includes: Each PCIe terminal device is combined with its corresponding server to form a basic PCIe system. Each basic PCIe system is initialized independently, assigned a corresponding ID and address space, and bound to a corresponding custom-designed chip for use.

7. The method according to claim 1, characterized in that, The method also includes: Each PCIe terminal device, together with the first PCIe bridge device, the second PCIe bridge device, and the root module with the ROOT port, forms a tree-like PCIe system. Each level of bridge device and all PCIe terminal devices in the tree-like PCIe system are assigned an independent and different bus, device ID, and address space.

8. A system for interconnecting multi-chip cluster servers, characterized in that, include: The configuration module is used to configure custom-designed chips for multiple PCIe terminal devices and the first PCIe bridge device, and to configure custom-designed switching chips for the PCIe ROOT port function module and the second PCIe bridge device. The connection module is configured to set up a custom-designed chip in each server cluster, establish a connection between each SOC server in the server cluster and the custom-designed chip through multiple PCIe terminal devices, and establish a connection between the second PCIe bridge device and each of the first PCIe bridge devices. The first interaction module is configured to respond to data interaction within the cluster, establish a first routing channel between two servers through the first PCIe bridge device, and interact with data through the first routing channel. as well as The second interaction module is configured to respond to data interaction between clusters, establish a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and perform data interaction through the second routing channel.

9. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images