Method for data communication between multiple computing nodes

Abstract

The present application relates to a method for data communication between multiple computing nodes, the method comprising: connecting the multiple computing nodes to the PCIe switching node via one of the multiple upstream ports of the PCIe switching node, respectively, wherein the PCIe switching node comprises multiple non-transparent bridges provided by the multiple computing nodes; initiating enumeration by the multiple computing nodes, establishing communication connections between the multiple computing nodes in the PCIe switching node; and in response to the establishment of the communication connection, respectively providing a first virtual network adapter in the multiple computing nodes, so that the multiple computing nodes perform the data communication through the first virtual network adapter, wherein the first virtual network adapter is formed by virtualizing the multiple non-transparent bridges.

Description

Technical Field

[0001] The present application relates to the field of data communication, and in particular, to a method for data communication between multiple computing nodes. Background Art

[0002] The theoretical bandwidth of Gigabit Ethernet is 1Gbps, and the theoretical bandwidth of 10Gbps is 10Gbps. With the increasing number of functions in vehicles (such as autonomous driving) and the subsequent generation and transmission of large amounts of data in vehicles, the traditional in-vehicle Ethernet technology faces challenges such as bandwidth. A larger theoretical bandwidth is needed to be applied to short-distance mass data communications in vehicles.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] An embodiment of the present application provides a method for data communication between multiple computing nodes, so as to realize short-distance high-bandwidth communication between computing nodes by virtualizing a PCIe device as a network adapter.

[0005] According to one aspect of the present application, a method for data communication between multiple computing nodes is provided, wherein the method includes: connecting the multiple computing nodes to the PCIe switching node via one of the multiple upstream ports of the PCIe switching node, respectively, wherein the PCIe switching node includes a plurality of non-transparent bridges set by the multiple computing nodes; initiating enumeration by the multiple computing nodes, establishing communication connections between the multiple computing nodes in the PCIe switching node; and in response to the establishment of the communication connection, respectively setting a first virtual network adapter in the multiple computing nodes, so that the multiple computing nodes perform the data communication through the first virtual network adapter, wherein the first virtual network adapter is formed by virtualizing the multiple non-transparent bridges.

[0006] In one or more embodiments of the present application, optionally, multiple PCIe domains corresponding to the multiple computing nodes are formed in the PCIe switch node by utilizing switch node division, and the multiple PCIe domains each have one upstream port among the multiple upstream ports and each have one non-transparent bridge among the multiple non-transparent bridges.

[0007] In one or more embodiments of the present application, optionally, the plurality of non-transparent bridges are set in the PCIe switch node by respectively establishing virtual end nodes in the plurality of PCIe domains and configuring configuration spaces of the virtual end nodes.

[0008] In one or more embodiments of the present application, optionally, establishing a communication connection between the multiple computing nodes includes: obtaining a storage space address of a non-transparent bridge by accessing a configuration space corresponding to a non-transparent bridge among the multiple non-transparent bridges.

[0009] In one or more embodiments of the present application, optionally, the method further includes: configuring the storage space to which the configuration space of the first virtual network adapter is mapped into a buffer for the data communication.

[0010] In one or more embodiments of the present application, optionally, the method also includes: connecting one or more end nodes to the PCIe switch node respectively through one downstream port among one or more downstream ports on the PCIe switch node, wherein the downstream port corresponds to one upstream port among the multiple upstream ports; establishing, through the enumeration initiated by the multiple computing nodes, a communication connection in the PCIe switch node between one of the multiple computing nodes connected to the upstream port and one of the one or more end nodes connected to the downstream port; and in response to the establishment of the communication connection, establishing a second virtual network adapter in the computing node and the end node respectively, so that the computing node and the end node perform the data communication through the second virtual network adapter.

[0011] In one or more embodiments of the present application, optionally, the first virtual network adapter is configured to a first network segment, and the second virtual network adapter is configured to a second network segment.

[0012] According to one aspect of the present application, a computer system is provided, comprising: at least one memory; at least one processor; and a computer program stored in the memory and executable on the processor, wherein the execution of the computer program on the processor results in the following operations: connecting a plurality of computing nodes to the PCIe switching node via one of a plurality of upstream ports of the PCIe switching node, respectively, wherein the PCIe switching node comprises a plurality of non-transparent bridges provided by the plurality of computing nodes; initiating enumeration by the plurality of computing nodes, establishing communication connections between the plurality of computing nodes in the PCIe switching node; and in response to the establishment of the communication connections, respectively providing a first virtual network adapter in the plurality of computing nodes, so that the plurality of computing nodes perform data communication through the first virtual network adapter, wherein the first virtual network adapter is formed by virtualizing the plurality of non-transparent bridges.

[0013] According to one aspect of the present application, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed by a processor, any one of the methods described above is implemented.

[0014] According to one aspect of the present application, a computer program product is provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, any one of the methods described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, aspects and advantages of the present application will become better understood when reading the following detailed description with reference to the accompanying drawings, in which the same or similar elements are represented by the same reference numerals. It should be noted that the drawings in the present application are only schematic and may not be drawn to scale or specific numbers. In the drawings:

[0016] Figure 1 A method for data communication between multiple computing nodes according to one or more embodiments of the present application is shown;

[0017] Figure 2 A schematic block diagram of a computer system. DETAILED DESCRIPTION

[0018] The present application is described more fully below with reference to the accompanying drawings in which the illustrative embodiments of the present application are illustrated. In the following detailed description of the embodiments, many specific details are set forth in order to provide a more thorough understanding of the disclosure of the present application. However, in one or more embodiments, well-known features are not described in detail to avoid making the description unnecessarily complicated. Where applicable, the embodiments of the present application and the features in the embodiments can also be combined with each other.

[0019] In the present application, terms such as "comprises", "includes", "has" and the like indicate that in addition to the units and steps directly and explicitly stated in the specification and claims, the technical solution described in the present application does not exclude the existence of other units and steps that are not directly or explicitly stated.

[0020] Unless otherwise specified, terms such as “first” and “second” do not indicate the order of elements in terms of time, space, size, etc., nor are they intended to limit any element to only a single element, but are merely used to distinguish between elements.

[0021] In the present application, a high-speed peripheral component interconnect switch node (PCIe (Peripheral Component Interconnect Express) Switch), a non-transparent / transparent bridge (Non-Transparent / Transparent Bridge), an end node (Endpoint, such as a computing node), etc. may be collectively referred to as PCIe devices, wherein one device may be contained in another device, for example, a PCIe switch node may contain a non-transparent / transparent bridge. Corresponding storage space may be allocated for these PCIe devices in a memory (such as a random access memory, RAM) (these PCIe devices thereby obtain addresses in the storage space, hereinafter referred to as storage space addresses), and the configuration space of the PCIe device may be mapped to the corresponding storage space. The configuration space is used to record information such as the device identification and classification code of the PCIe device and to configure the function of the PCIe device (for example, by writing to the base address register (BAR)).

[0022] The computing node referred to in this application can be configured to be connected to a PCIe switch node in communication (for example, via an upstream port and a downstream port on the PCIe switch node), and may include but is not limited to a system on chip, a chipset, a central processing unit, etc.

[0023] Those skilled in the art should appreciate that the above-mentioned devices or nodes may be implemented as hardware or software (eg, virtual end nodes) or a combination of hardware and software, unless otherwise explicitly or implicitly stated in the context.

[0024] Figure 1 A method 100 for data communication between multiple computing nodes according to one or more embodiments of the present application is shown. The method 100 includes steps 110, 120, and 130.

[0025] In step 110, multiple computing nodes are connected to the PCIe switching node via one of the multiple upstream ports of the PCIe switching node, respectively. By integrating the root complex function in the multiple computing nodes, a PCIe bus (e.g., a virtual PCIe bus), a PCI-PCI bridge (or generally referred to as a P2P bridge) device, etc., can be provided for the multiple computing nodes to be used for connecting the upstream port of the PCIe switching node. Accordingly, in the PCIe switching node, a P2P bridge (e.g., a virtual P2P bridge, which may include a non-transparent bridge and a transparent bridge) is also provided for connecting the computing node with one or more end nodes. Since the non-transparent bridge provides, for example, address conversion (or referred to as address translation, which is implemented by using the base address register described below) between two root complexes in different storage spaces, a computing node can be used to set a non-transparent bridge to implement data communication between a computing node of the integrated root complex and other root complexes (e.g., one or more other computing nodes of the integrated root complex), and correspondingly, a transparent bridge can be used to implement data communication between the root complex and the end node connected to the root complex via the PCIe switching node (without the above-mentioned address conversion).

[0026] In one or more embodiments, multiple PCIe domains corresponding to the multiple computing nodes can be formed (e.g., virtually) in a PCIe switching node by utilizing a switching node partitioning technique, and each PCIe domain can provide an upstream port and one or more downstream ports corresponding to the computing node. Specifically, a computing node acting as a root complex can write to the port configuration register of the PCIe switching node to set the PCIe domain identifier, port mode (e.g., upstream, downstream), and downstream port relationship, etc. In addition, the upstream port and the downstream port can also be modified, for example, an upstream port / downstream port can be added to a PCIe domain or an upstream port / downstream port can be removed from the PCIe domain. In this way, multiple discrete switching nodes can be replaced with multiple virtual switching nodes, so as to flexibly configure, for example, the upstream port required to connect one or more computing nodes, and can also save costs, reduce power consumption, etc.

[0027] In one or more embodiments, virtual end nodes may be established in multiple PCIe domains respectively and the configuration space of the virtual end nodes may be configured to establish a non-transparent bridge in a PCIe switching node. Specifically, in the configuration space corresponding to the virtual end node set by the computing node connected to this PCIe domain, there is a base address register (BAR) that can be configured by the computing node, and the BAR numbered 0 (hereinafter referred to as BAR0) may be configured as a control field for implementing an NTB function (e.g., address translation), and the control field indicates the PCIe connection status, the number of doorbell registers, the offset address of the starting doorbell register, the number of scrachpad registers for non-transparent bridges, the offset address of the starting scrachpad register, the capacity of the shared window, the offset address of the first shared window, etc. Correspondingly, BAR1 may indicate, for example, the offset address of the scratchpad register for another non-transparent bridge, etc., and BAR2 may be configured as, for example, a doorbell register, a shared window, etc. The doorbell register may be used to generate, for example, an interrupt to a computing node when, for example, a new register write operation occurs, and the scratchpad register may be used to transfer data (for example, the address of a PCIe device recorded in BAR1 in its configuration space) between, for example, BAR0 of a non-transparent bridge in a first PCIe domain and BAR1 of a non-transparent bridge in a second PCIe domain, so that data communication between computing nodes may be achieved by accessing the storage space address in a corresponding storage space in a memory (for example, RAM) to which the address of the PCIe device in the PCIe domain (i.e., the address in the configuration space) recorded in the BAR is mapped.

[0028] In step 120, the multiple computing nodes may initiate enumeration of the downstream (e.g., the PCIe switching node in the present application), and establish a communication connection between the multiple computing nodes in the PCIe switching node. Specifically, each computing node may enumerate the downstream PCIe devices connected to itself by combining the bus number, device number, and function number. The bus number indicates the number of the bus where the PCIe device is located in the PCIe topology structure. For example, the bus number between the P2P bridges in the first computing node may be 0, and the bus number between the P2P bridge / non-transparent bridge connecting the upstream port and the downstream port / non-transparent bridge port in the first PCIe domain may be 1, and so on. The device number indicates the number of the device in, for example, a computing node or a switching node. For example, a computing node may include three virtual P2P bridges numbered 0, 1, and 2, respectively. The function number indicates the number assigned to each PCIe function within the same PCIe device. If the device is configured to implement a function, it is usually 0. Through the above numbering, the PCIe computing node can determine the layout (e.g., connection relationship) of the downstream (e.g., PCIe switching node) PCIe devices (e.g., end nodes, non-transparent bridges connected to other PCIe domains) and their addresses within the PCIe domain, thereby establishing a communication connection between computing nodes via the non-transparent bridge.

[0029] In one or more embodiments, the establishment of a communication connection between the multiple computing nodes may include: obtaining the storage space address of a non-transparent bridge among the multiple non-transparent bridges by accessing the configuration space corresponding to the non-transparent bridge. When the multiple computing nodes establish a communication connection between each other, one of the computing nodes may read the configuration space address of the PCIe device (e.g., a non-transparent bridge in another PCIe domain) recorded in the BAR in the corresponding PCIe domain and then obtain the storage space address of the PCIe device in the memory (e.g., RAM) mapped to the address, thereby accessing the storage space of the non-transparent bridge in another PCIe domain, for example, to perform operations such as reading and writing. Accordingly, other computing nodes may operate as above, thereby establishing a communication connection between the multiple computing nodes that can realize reading and writing data through the non-transparent bridge.

[0030] In step 130, in response to the establishment of the above-mentioned communication connection, virtual network adapters (for example, first virtual network adapters are set for each) can be respectively set on the plurality of computing nodes, so that the plurality of computing nodes perform data communication through the virtual network adapters. Specifically, a PCIe driver can be generated, for example, using data structures such as pci_driver, pci_bus_type, pci_bus, pci_slot, and pci_host_bridge in the Linux system kernel, and the PCIe driver can be used to virtualize a PCIe device (for example, a non-transparent bridge or an end node described below) into a virtual network adapter, which is provided at the data link layer to support network communication based on the TCP / IP protocol. Then, the virtual network adapter can be added to the computing node by, for example, the ifconfig tool under the Linux system, so as to set the virtual network adapter on the computing node.

[0031] In one or more embodiments, the storage space (e.g., a portion or all of the storage space of the non-transparent bridge) to which the configuration space of the virtual network adapter (e.g., the first virtual network adapter) is mapped can be configured as a buffer for data communication. For example, as described above, the address in the storage space to which the virtual network adapter is mapped can be determined according to the address of the virtual network adapter in the PCIe domain recorded in the BAR in the configuration space. The buffer can be divided into a data receiving buffer and a data sending buffer for the computing node to read and write.

[0032] In one or more embodiments, in the manner described above, one or more end nodes can be connected to the PCIe switch node through one of the one or more downstream ports on the PCIe switch node, and the downstream port is connected to one of the multiple upstream ports through, for example, a transparent bridge in the PCIe switch node. Then, through the enumeration initiated by the multiple computing nodes, a communication connection is established in the PCIe switch node between one of the multiple computing nodes connected to the upstream port and one of the one or more end nodes connected to the downstream port. And in response to the establishment of the communication connection, a second virtual network adapter is established in the computing node and the end node, respectively, so that the computing node and the end node perform the data communication through the second virtual network adapter.

[0033] In one or more embodiments, the first virtual network adapter can be configured to the first network segment, and the second virtual network adapter can be configured to the second network segment, so that TCP / IP communication without network address translation can be achieved between computing nodes in the same network segment or between computing nodes and end nodes connected to them through PCIe switching nodes. For situations in different network segments, such as between a computing node and an end node connected to another computing node through a PCIe switching node, TCP / IP communication based on network address translation can be achieved.

[0034] Although Figure 1 The steps in the embodiment are presented and described sequentially, but a person of ordinary skill will appreciate that some or all of the steps may be performed in a different order, may be combined or omitted, and some or all of the steps may be performed in parallel, and additional steps may be further performed. Therefore, the scope of the present disclosure should not be considered to be limited to Figure 1 A specific arrangement of the steps shown in .

[0035] Figure 2 is a schematic block diagram of a computer system. Figure 2 As shown, the computer system 200 includes at least one memory 210 (e.g., a non-volatile memory such as a flash memory, a ROM, a hard disk drive, a magnetic disk, an optical disk, etc.), at least one processor 220, and a computer program 230. The memory 210 stores the computer program 230 executable by the processor 220. The processor 230 is configured to run the computer program 230 stored on the memory 210. The above-mentioned operation can be realized by running the computer program stored on one or more memories on one or more processors (e.g., a method in which multiple processors run the computer program in cooperation or a method in which a single processor runs the computer program alone). Figure 1 One or more steps or operations included in the method.

[0036] According to another aspect of the present application, a computer-readable storage medium is provided on which instructions are stored. When the instructions are executed by a processor, the above-mentioned Figure 1 One or more steps or operations included in the method.

[0037] According to another aspect of the present application, a computer program product is also provided, the computer program product comprising computer instructions, the computer instructions can be executed by a processor to achieve the above Figure 1 One or more steps or operations included in the method.

[0038] The processor referred to in this application may be an integrated circuit chip having the ability to process signals. Figure 1One or more steps or operations included in the method may be performed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0039] The computer-readable storage medium referred to in the present application includes various types of computer storage media, which can be any available medium that a general or special-purpose computer can access. For example, a computer-readable storage medium can include RAM, ROM, EPROM, E2PROM, register, hard disk, removable disk, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or can be used to carry or store a desired program code unit with an instruction or data structure form and can be accessed by a general or special-purpose computer, or a general or special-purpose processor. Any other temporary or non-temporary medium. The above combination should also be included in the protection scope of the computer-readable storage medium. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium can be integrated into a processor.

[0040] It should be noted that the embodiments described in this application are intended to make the disclosure herein comprehensive and complete, and are not intended to limit the scope of the claimed subject matter. Those skilled in the art may think of other feasible changes or replacements based on the technical scope disclosed in this application, and such changes or replacements are all included in the protection scope of this application. Those skilled in the art will know that the above description and examples are provided only for the convenience of explanation and example, and the proposed description and examples are not intended to cover all aspects of the application or limit the application to the disclosed precise form. The technical solutions recorded in this application can be implemented in different forms without departing from the spirit and scope of the application.

Claims

1. A method for data communication between multiple computing nodes, the method comprising: Connecting the plurality of computing nodes to the PCIe switching node via one of the plurality of upstream ports of the PCIe switching node, respectively, wherein the PCIe switching node includes a plurality of non-transparent bridges provided by the plurality of computing nodes; connecting one or more end nodes to the PCIe switch node through one of the one or more downstream ports on the PCIe switch node, wherein the downstream port corresponds to one of the plurality of upstream ports; Initiating enumeration through the plurality of computing nodes, establishing communication connections between the plurality of computing nodes in the PCIe switch node, and establishing a communication connection between one of the plurality of computing nodes connected to the upstream port and one of the one or more end nodes connected to the downstream port; and In response to the establishment of the communication connection, a first virtual network adapter is set on each of the multiple computing nodes, so that the multiple computing nodes perform the data communication through the first virtual network adapter, and a second virtual network adapter is established on the computing node and the end node, so that the computing node and the end node perform the data communication through the second virtual network adapter, wherein the first virtual network adapter is formed by virtualizing the multiple non-transparent bridges.

2. The method of claim 1, wherein: A plurality of PCIe domains corresponding to the plurality of computing nodes are formed in the PCIe switch node by utilizing switch node division, and the plurality of PCIe domains each have one upstream port among the plurality of upstream ports and one non-transparent bridge among the plurality of non-transparent bridges.

3. The method of claim 2, wherein: The plurality of non-transparent bridges are arranged in the PCIe switching node by respectively establishing virtual end nodes in the plurality of PCIe domains and configuring the configuration space of the virtual end nodes.

4. The method of claim 1, wherein: Establishing communication connections between the plurality of computing nodes includes: obtaining a storage space address of a non-transparent bridge by accessing a configuration space corresponding to a non-transparent bridge among the plurality of non-transparent bridges.

5. The method of claim 1, further comprising: The storage space to which the configuration space of the first virtual network adapter is mapped is configured as a buffer for the data communication.

6. The method of claim 1, wherein: The first virtual network adapter is configured to a first network segment, and the second virtual network adapter is configured to a second network segment.

7. A computer system comprising: at least one memory; at least one processor; as well as A computer program stored on the memory and executable on the processor, the execution of the computer program on the processor causing the following operations: Connecting a plurality of computing nodes to the PCIe switching node via one of a plurality of upstream ports of the PCIe switching node, respectively, wherein the PCIe switching node includes a plurality of non-transparent bridges provided by the plurality of computing nodes; connecting one or more end nodes to the PCIe switch node through one of the one or more downstream ports on the PCIe switch node, wherein the downstream port corresponds to one of the plurality of upstream ports; Initiating enumeration through the plurality of computing nodes, establishing communication connections between the plurality of computing nodes in the PCIe switch node, and establishing a communication connection between one of the plurality of computing nodes connected to the upstream port and one of the one or more end nodes connected to the downstream port; and In response to the establishment of the communication connection, a first virtual network adapter is set on each of the multiple computing nodes, so that the multiple computing nodes perform data communication through the first virtual network adapter, and a second virtual network adapter is established on the computing node and the end node, so that the computing node and the end node perform the data communication through the second virtual network adapter, wherein the first virtual network adapter is formed by virtualizing the multiple non-transparent bridges.

8. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product, comprising computer instructions, characterized in that: The computer instructions implement the method according to any one of claims 1 to 6 when executed by a processor.

Images