An inter-device interconnection method compatible with a PCIe structure and related devices

Abstract

The application provides an interconnection method between devices compatible with a PCIe structure and related devices, and the interconnection method is applied to a heterogeneous system, the heterogeneous system comprising a host and a plurality of devices connected to the host, and the method comprises: a first device changing a type of a data path of the first device according to a preset control character received from a second device or the host; and the first device after the type is changed and the second device realizing interconnection; wherein the preset control character is an instruction conforming to a PCIe communication protocol and capable of controlling the first device to change the type of the data path of the first device, and the type of the data path is a root complex (RC) type or an endpoint device (EP) type. The application can realize interconnection between devices by switching the device type, and can reduce data transmission delay of interconnection between devices in a heterogeneous computing system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular to a method for interconnecting devices compatible with PCIe (Peripheral Component Interconnect Express, PCIe for short) structure and related devices. BACKGROUND

[0002] With the rapid development of general high-performance computing systems, artificial intelligence and multimedia image processing, the demand for data centers and cloud computing in modern society is increasingly dependent. The core engine of the data center and cloud computing, high-performance general-purpose computing chips, also have further requirements. Today, the rapidly expanding data and computing volume has far exceeded the capacity of traditional CPUs.

[0003] To make up for the lack of CPU computing power, a common solution is to use heterogeneous computing systems with co-processing chips to improve computing performance. Common co-processing chips have multiple options such as GPU (English: Graphics Processing Unit, GPU for short, Chinese: graphics processor), ASIC (English: Application Specific Integrated Circuit, ASIC for short, Chinese: Application Specific Integrated Circuit), FPGA (English: Field Programmable Gate Array, FPGA for short, Chinese: Field Programmable Gate Array).

[0004] And how to further expand and improve the performance of heterogeneous computing, reduce the data transmission delay required by the computing-intensive workload, has become the main problem. SUMMARY

[0005] The present application provides a method for interconnecting multiple devices compatible with PCIe structure and related devices, to solve the problem of lack of interconnection path between devices in a heterogeneous computing system based on PCIe topology, resulting in excessive data transmission delay, to realize the interconnection between devices.

[0006] The present application provides a method for interconnecting devices compatible with PCIe structure, applied to a heterogeneous system, the heterogeneous system comprising a host and a plurality of devices connected to the host, the interconnection method comprising:

[0007] The first device changes the type of its data path according to the preset control character received from the second device or the host;

[0008] The first device changes the type of its data path according to the preset control character received from the second device or the host;

[0009] The preset control character is an instruction conforming to the PCIe communication protocol and capable of controlling the first device to change the type of the data path of the first device, and the type of the data path is a root complex (RC) type or an endpoint device (EP) type.

[0010] According to the PCIe structure compatible device interconnection method, the first device changes the type of the data path of the first device according to the received preset control character sent by the second device, and the method comprises the following steps of:

[0011] The second device sends a training character containing a second preset link code to the first device.

[0012] The first device switches the type of the data path of the first device which first receives the continuous training character containing the second preset link code.

[0013] The first device sends a training character containing a first preset link code to the second device through the data path after the type switching, so as to realize the interconnection of the first device and the second device.

[0014] The first preset link code is used to define the type of the data path of the first device, the second preset link code is used to define the type of the data path of the second device, and the type of the data path is a root complex (RC) type or an endpoint device (EP) type.

[0015] According to the PCIe structure compatible device interconnection method, after the first device sends a training character containing a first preset link code to the second device through the data path after the type switching, the method comprises the following steps of:

[0016] When the remaining data paths of the first device receive the continuous training character containing the second preset link code, the types of the remaining data paths of the first device are switched according to the type of the data path.

[0017] According to the PCIe structure compatible device interconnection method, before the second device sends a training character containing a second preset link code to the first device, the method comprises the following steps of:

[0018] The timeout timing mechanism is set for the first device and the second device, so as to ensure that all the data paths of the first device are re-established after the type switching.

[0019] According to the PCIe structure compatible device interconnection method, the steps of working of the timeout timing mechanism comprise the following steps of:

[0020] The host confirms whether the peer-to-peer interconnection between the devices is successful by detecting a link status register in the device;

[0021] If the link status is always in the False state within a preset time, it is considered that the peer-to-peer interconnection between the devices is unsuccessful;

[0022] The timing value is reset, and the establishment of the peer-to-peer interconnection between the devices is reattempted;

[0023] The host locates and monitors the position and state information between the devices by a preset bus function device code, and changes or monitors the link status register between the devices in real time.

[0024] According to the interconnection method between devices compatible with a PCIe structure, when the data path of the first device is of an RC type and the data path of the second device is of an RC type structure, the interconnection method comprises the following steps:

[0025] The second device sends a training character containing a second preset link code to the first device;

[0026] The first device switches the type of the data path, which first receives the continuous training character containing the second preset link code, from the RC type to the EP type;

[0027] The first device sends a training character containing a first preset link code to the second device through the data path after the type switching, so as to realize the interconnection between the second device and the first device.

[0028] According to the interconnection method between devices compatible with a PCIe structure, when the data path of the first device is of an RC type and the data path of the second device is of an RC type structure, the interconnection method comprises the following steps:

[0029] The second device sends a training character containing a second preset link code to the first device;

[0030] The first device switches the type of the data path, which first receives the continuous training character containing the second preset link code, from the RC type to the EP type;

[0031] The first device sends a training character containing a first preset link code to the second device through the data path after the type switching, so as to realize the interconnection between the second device and the first device.

[0032] According to the PCIe-compatible interconnection method of the present invention, in a PCIe-compatible structure where the data path of the first device is of type RC and the data path of the second device is of type EP, or in a PCIe-compatible structure where the data path of the first device is of type EP and the data path of the second device is of type RC, the PCIe link is re-established, including:

[0033] The second device sends training characters containing the second link encoding to the first device through the sending end of the data path;

[0034] After receiving the training characters, the first device identifies the type of the second device based on the second link encoding and then configures its own first link encoding.

[0035] The first device sends training characters containing the first link code to the second device through the sending end of the data path, so as to inform the second device of the first link code;

[0036] After the second device receives the training characters and confirms the first link encoding, it completes the establishment of the PCIe link between the first device and the second device.

[0037] According to the PCIe-compatible inter-device interconnection method of the present invention, the first device changes the type of its own data path based on a preset control character received from the host, including:

[0038] The host assigns a bus function device code to each device to determine the coordinate position of each device in the entire mechanism system;

[0039] Based on the preset bus function device code, the host can control the switching of the type of peer-to-peer interconnected devices.

[0040] This invention also provides a PCIe-compatible inter-device interconnect system, including a CPU, a root complex (RC), a memory, a switch, and multiple devices. The CPU is connected to the memory, the switch, and the multiple devices via the root complex. The devices include a first device and a second device.

[0041] The second device sends training characters containing a second preset link code to the first device; the first device switches the type of the data path that first receives consecutive training characters containing the second preset link code; the first device sends training characters containing a first preset link code to the second device through the data path after the type switch, so as to realize the interconnection between the first device and the second device;

[0042] The first preset link code is used to define the type of the first device, and the second preset link code is used to define the type of the second device.

[0043] The application further provides an interconnection chip between devices compatible with a PCIe structure, applied to a heterogeneous computing system.

[0044] The second chip sends a training character containing a second preset link code to the first chip;

[0045] The first chip switches the type of a data path that first receives a continuous training character containing a second preset link code;

[0046] The first chip sends a training character containing a first preset link code to the second chip through the data path after the type switching, so as to realize the interconnection between the first chip and the second chip.

[0047] The first preset link code is used to define the type of the data path of the first chip, and the second preset link code is used to define the type of the data path of the second chip.

[0048] The application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0049] The application further provides a non-transitory computer readable storage medium, which stores a computer program executable on a processor to realize the steps of the interconnection method between devices compatible with a PCIe structure.

[0050] The interconnection method between devices compatible with a PCIe structure and related devices provided by the application can realize the interconnection between devices by switching the device type, so as to realize the reading and writing of data between devices, reduce the data transmission delay of the interconnection between devices in a heterogeneous computing system, expand the data bandwidth of multi-device interconnection, and further eliminate the bottleneck of a heterogeneous computing system. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to make the technical solutions in the present application or prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0052] Figure 1 is a schematic diagram of a heterogeneous computing system provided by the prior art;

[0053] Figure 2 is one of the flow schematic diagrams of the device interconnection method compatible with the PCIe structure provided by the present application;

[0054] Figure 2a is another of the flow schematic diagrams of the device interconnection method compatible with the PCIe structure provided by the present application;

[0055] Figure 3 is a conversion schematic diagram of the PCIe interconnection topology provided by the present application;

[0056] Figure 4 is a schematic diagram of the PCIe link establishment provided by the present application;

[0057] Figure 5 is a schematic diagram of the device interconnection of the RC / RC structure provided by the present application;

[0058] Figure 6 is a schematic diagram of the device interconnection of the EP / EP structure provided by the present application;

[0059] Figure 6a is a schematic diagram of the Timeout timing mechanism provided by the present application;

[0060] Figure 7a is a structural schematic diagram of the device interconnection system compatible with the PCIe structure provided by an embodiment of the present application;

[0061] Figure 7b is a structural schematic diagram of the device interconnection system compatible with the PCIe structure provided by another embodiment of the present application;

[0062] Figure 8 is a flowchart of the device interconnection method compatible with the PCIe structure provided by the present application;

[0063] Figure 9 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0065] Artificial intelligence (AI) has three elements: algorithm, computing power, and data. Among them, computing power is ultimately provided by the underlying chip. According to the composition of the computing chip, it can be divided into homogeneous computing and heterogeneous computing, as follows:

[0066] Homogeneous computing: a computing method that uses computing units of the same type of instruction set and architecture to form a system.

[0067] Heterogeneous computing: a computing method that uses computing units of different types of instruction sets and architectures to form a system. Common computing unit categories include CPU, GPU, ASIC, FPGA, etc.

[0068] For example: insert a GPU (Graphics Process Unit, GPU for short, graphics processing unit) into the PCIe (Peripheral Component Interconnect Express, PCIe for short, high-speed peripheral component interconnect) socket of the mainboard. When running the program, the CPU can be regarded as a commander, that is, the host, and the GPU that completes a large amount of calculation is a computing device, that is, the device. The CPU and the GPU are connected through the PCIe bus for transmitting instructions and data.

[0069] Figure 1 It is a schematic diagram of a heterogeneous computing system provided by the prior art, as shown in the figure. The heterogeneous computing system shown in the figure, such as a CPU and a multi-device computing system, is interconnected through a high-speed IO bus such as PCIe.

[0070] The data transmission between CPU and Device, Device and Device all follow the PCIe protocol specification. Host and Device, Device and Device share IO bus bandwidth. The general heterogeneous computing system 100, CPU 101 is interconnected with system memory (System Memory) 103, Swich (Switch) 104, Swich 105 and multiple Devices (Device0 106, Device1 107, Device2 108, Device3 109) through RC (Root Complex, RC for short) 102, and the entire heterogeneous system follows the PCIe bus protocol topology.

[0071] In the above, RC is the root of the PCIe bus, and RC connects the bus, memory and processor to the bus channel. EP (Endpoint, EP for short) is the general term for PCI / PCIe devices, such as PCIe network cards, PCIe graphics cards, etc. Switch: simply a PCIe switch, used to expand the PCIe interface.

[0072] Based on the above Figure 2a heterogeneous computing system 100, each Device (Device0 106, Device1 107, Device2 108, Device3 109) and CPU 01 are provided with independent data transmission paths, such as transmission path 110 between Device1 107 and CPU 101, and transmission path 112 between Device2 108 and CPU 101. Based on the existing PCIe topology, there is no independent data path between Device and Device, such as no independent data path between Device1 107 and Device2 108, and their data path 111 is highly shared with CPU transmission path 110 and 112. Moreover, the data exchange path between Device1 107 and Device2 108 is extremely long, and the data exchange between Device1 107 and Device2 108 must go through RC 102 and swtich 104.

[0073] Therefore, the data transmission delay between devices in a heterogeneous system based on a PCIe topology is large, and the structure of sharing data paths with a CPU has a significant impact on the data transmission bandwidth between devices, so that the data transmission bandwidth between devices becomes the bottleneck of the overall heterogeneous computing system. And under the strict PCIe topology-based multi-device interconnection system, an additional off-chip switch is generally needed to assist in connection. In this topology, not only does it increase system complexity and hardware cost, but it also cannot achieve efficient two-by-two interconnection structure under multi-device.

[0074] Generally, the two-by-two interconnection structure under the PCIe topology-based multi-device interconnection needs to pre-determine the ID and position of each device in the system, and then the Host initializes and configures each device according to the obtained ID and position information, and then the connection between devices can start to establish connection. Once a device joins or exits the system, the ID number and position information of each device need to be recalculated and allocated by the Host. This method greatly limits the flexibility of multi-device interconnection and the difficulty of software implementation.

[0075] Therefore, the present application proposes a more flexible two-by-two interconnection implementation method between devices based on autonegotiation under the compatible PCIe topology structure, which realizes the interconnection between devices under the compatible PCIe topology structure such as RC / EP interconnection structure, and also supports the interconnection between devices under the irregular RC / RC and EP / EP interconnection topology structure.

[0076] Figure 2 It is one of the flowcharts of the device interconnection method compatible with the PCIe structure provided by the present application, as shown in the figure. A device interconnection method compatible with the PCIe structure is applied to a heterogeneous system, the heterogeneous system comprising a host and a plurality of devices connected to the host, the interconnection method comprising:

[0077] Step 200, the first device changes the type of its data path according to the preset control character received from the second device or the host.

[0078] Step 201, the first device changes the type of its data path according to the preset control character received from the second device or the host.

[0079] The preset control character is an instruction that conforms to the PCIe communication protocol and can control the first device to change the type of its own data path, and the type of the data path is a root complex RC type or an endpoint device EP type.

[0080] Figure 2a Figure 2 is a flowchart of a second embodiment of the method for interconnecting devices compatible with PCIe structure according to the present application, as shown in the figure. The first device changes the type of its data path according to the preset control instruction received from the second device, including:

[0081] In step 210, the first device sends a training character containing a first preset link code to the second device.

[0082] Optionally, the first preset link code can be set as PAD link number.

[0083] In step 220, the second device switches the type of the data path that first receives the continuous training character containing the first preset link code.

[0084] In step 230, the second device sends a training character containing a second preset link code to the first device through the data path after type switching, so as to realize the interconnection between the first device and the second device.

[0085] Optionally, the second preset link code can be set as non-PAD link number.

[0086] In the above embodiment, the first preset link code is used to define the type of the first device, and the second preset link code is used to define the type of the second device; and the type of the data path is RC type or EP type.

[0087] It should be noted that the first device and the second device each have one or more data paths.

[0088] Optionally, the first preset link code can be defined as the type of the data path being EP type, and the second preset link code can be defined as the type of the data path being RC type.

[0089] Therefore, the method for interconnecting devices compatible with PCIe structure according to the present application can be applied to a heterogeneous computing system, is suitable for the architecture of CPU and multiple devices, especially the interconnection between multiple devices, can reduce the data transmission delay of the interconnection between multiple devices in the heterogeneous computing system, expand the data bandwidth of the interconnection between multiple devices, and further eliminate the bottleneck of the heterogeneous computing system.

[0090] Specifically, the Device interconnection structure described in the present application can avoid the introduction of multiple switches to increase the complexity and cost of the system, and reduce the cost of the system.

[0091] Specifically, under the compatibility of complete PCIe topology and rules, the complexity of realizing the interconnection between multiple devices is reduced by introducing a Host programing mode, and the implementation difficulty between software and hardware is balanced.

[0092] Specifically, the device interconnection system compatible with the PCIe structure does not need to determine the ID and position information of each device in advance, and the connection of each device completely relies on hardware adaptation, which simplifies the software implementation difficulty and greatly enhances the flexibility of the system.

[0093] The specific structure of the device interconnection method compatible with the PCIe structure will be described below.

[0094] Figure 3 The PCIe interconnection topology conversion schematic diagram provided by the present application is shown in the figure. In the non-conventional RC / RC or EP / EP interconnection structure, the two interconnection devices realize the autonegotiation mechanism in the link establishment process to adapt the Device Type between RC / RC structure and EP / EP structure. As shown in figure 300, the non-conventional RC / RC, EP / EP interconnection structure is finally converted into RC / EP structure and EP / RC structure, thereby meeting all the requirements of the PCIe framework.

[0095] Specifically, the RC / RC structure of 301 is converted into the RC / EP structure of 302 or the EP / RC structure of 303 through the adaptive device type switching to realize the interconnection between devices. Because according to the existing PCIe architecture rules, the RC / RC structure and the EP / EP structure cannot be interconnected.

[0096] Figure 4 The PCIe link establishment schematic diagram provided by the present application is shown in the figure. In the traditional PCIe link establishment process, two interconnection devices (such as RC 409 and EP 408) define their own type by recognizing the PAD link number of the opposite Training symbol. Lane_a0 represents one data path on the RC 409 side, and Lane_b0 represents one data path on the EP 408 side.

[0097] Generally, in the conventional PCIe topology, the RC 409 first sends a training symbol containing a link number to the opposite end EP 408 through the sending end TX 401 of the data path, informing the opposite end EP 408 of the link number of the RC 409, and the EP 408 passively receives the training symbol through the receiving end RX 410 of the data path.

[0098] The EP 408 configures its own link number according to the link number sent by the RC 409, and feeds back to the RC 409 through the sending end TX 411 of the data path, that is, sends a training symbol containing its own configured link number to the RC 409, to inform the RC 409 of the link number configured by the EP 408. After that, the RC 409 passively receives the link number sent by the EP 408 through the receiving end RX 412 of the data path and confirms the data, and the interconnection of the RC 409 and the EP 408 in the PCIe topology is preliminarily completed.

[0099] Figure 5 The RC / RC structure of the device interconnection provided by the application is shown in the figure. In the RC 509 and RC 508 interconnection structure, the RC 508 switches the type of the data path lane_b1 501, which first receives the continuous training symbol 502 containing the second preset link number (non-PAD link number), to the data path type of the EP attribute.

[0100] Until all the data paths (such as lane_b1 701, lane_b0 703, lane_b2 704, and lane_b3 705) of the RC attribute on the side of the RC 508 are all converted to the data path type of the EP attribute, that is, the switching of the device type from the RC type to the EP type is completed, and after the Timeout timing time of the pre-set Training state machine, the PCIe Link is re-established, and the PCIe Link establishment is as shown in Figure 4

[0101] Therefore, after the Auto negotiation mechanism, the two Device (RC 509 / RC 508) are converted from the RC / RC interconnection structure to the RC / EP interconnection topology, that is, the RC type of the RC 508 is switched to the EP type. ​

[0102] It should be noted that, in order to identify the RC and the EP, the embodiment of the present application defines that the non-PAD link number is sent for the RC type definition, and the PAD link number is sent for the EP type definition, but the present application is not limited thereto, and any definition mode that is beneficial to identifying the RC and the EP type can be used.

[0103] Figure 6 is a schematic diagram of the interconnection between the devices of the EP / EP structure provided by the present application, as shown in the figure. Under the EP 609 and EP 608 interconnection structure, the EP 608 sends the training symbol containing the PAD link number to the EP 609 through the data path 602.

[0104] The EP 609 switches the type of the data path Lane_a2 604 that first receives the continuous training symbol containing the PAD link number to the RC type, and sets the link number of all the data paths (such as the Lane_a0 603, the Lane_a1 601, and the Lane_a3 605) on the same side as the data path to the default value, which can be controlled by a register and can be configured during power-on initialization. Then, the EP 609 sends the training symbol 606 containing the non-PAD link number to the data path Lane_b2 607 of the EP 608 through the data path 606 that has been switched to the RC type.

[0105] At the same time, the other data paths (such as the Lane_a0 603, the Lane_a1 601, and the Lane_a3 605) of the EP 609 that have not received the continuous PAD link number sent by the EP 608 continue to maintain the PAD link number sent to the other data paths of the EP 609, and the other data paths maintain the data path type of the EP attribute of the other data paths until the other data paths also receive the continuous PAD link number sent by the EP 608 and switch the data path type of the other data paths from the EP type to the RC type.

[0106] It should be noted that the switching of the data path type described above is based on the pre-set Timeout timing mechanism, the Device type (device type) of the switched data path is maintained, and the PCIe link establishment is re-performed.

[0107] The Timeout timing mechanism described above is further described as follows:

[0108] The introduction of a timeout mechanism further prevents dead locks between links. The host monitors the link-up status register in the device to confirm the success of the peer-to-peer interconnection. If the link-up status of a peer-to-peer link remains False for a certain period, it is considered that auto-negotiation has entered a dead lock under the current state. A new timeout value needs to be set, and auto-negotiation needs to be retried to establish the peer-to-peer interconnection. The introduction of PCIe BDF numbers (Bus number, Device number, Function number) allows the host to easily and quickly locate and monitor the position and status information of each device in the system. It also allows for real-time modification or monitoring of the peer-to-peer interconnection status control registers between devices in heterogeneous systems.

[0109] The host first generates random, unequal values ​​that conform to the auto-negotiation rules, based on a certain algorithm. Then, the host can use the enumerated BDF numbers (Bus number, Device number, Function number) to locate the coordinates of multiple devices at the system application level, distinguishing them in detail as Device 0, Device 1, Device 2, and Device 3. A pre-generated random timeout value is then set sequentially for each device with a different BDF number. For example... Figure 6a As shown, the Host read / write control command reaches the Device's internal / external bus control unit 612a via the external central control bus 618a. Then, the received random timeout value is forwarded to the Tr timer in each peer interconnection unit EP 616a, EP 614b, and 615c via the Device's internal data control bus. The Tr timer, combined with the timeout value set by the host and the free-running clock generated by the PLL (phase-locked loop) of the Device's SMU (System Management Unit), performs timing operations, thus completing the entire timeout mechanism.

[0110] Therefore, this invention, by introducing an auto-negotiation (adaptive) mechanism, enables peer-to-peer interconnection between devices and is compatible with PCIe link establishment rules and topology. Furthermore, the timeout mechanism not only prevents deadlocks between RC / RC and EP / EP during auto-negotiation but also indirectly controls the type switching of EP / RC.

[0111] Another embodiment of the present invention provides a method for interconnecting devices compatible with PCIe architecture, namely, a method for directly controlling the establishment of peer-to-peer interconnection links through a top-down structure using a Host software layer.

[0112] Typically, in the traditional PCIe topology, such as Figure 7b As shown, Host RC 302 connects to each Device0 / Device1 / Device2 / Device3 via Switch 304 or directly through the external central control buses 318a / 318b / 318c / 318d. After establishing the transmission link, Host RC enumerates each Device to assign a BFD (Bus, Function, Device) Number, which is a preset bus function device code. This determines the coordinate position of each Device in the entire heterogeneous system. The system software layer can identify each Device in the system through this BFD number and control the attributes of any peer-to-peer interconnected control unit in the heterogeneous system as a whole.

[0113] For example, Device0 / Device1 / Device2 / Device3 are multiple devices with completely equivalent functions. Their initial peer-to-peer interconnect control units have a default attribute (or "type") of EP. The software layer can control the peer-to-peer interconnect units 313a / 314a / 315a in Device0306a to maintain the EP attribute by reading the BFD number assigned to each device. However, for the control units 313b / 314d / 315c connected to the 313a / 314a / 315a peer-to-peer interconnect units, the host can directly control the attributes of each device's internal registers according to certain rules or algorithms, switching them to RC. This establishes an RC / EP, EP / RC peer-to-peer interconnect bus data link conforming to the PCIe topology.

[0114] Figure 7ais an embodiment of the present application provides the structure diagram of the interconnection system between devices compatible with PCIe structure, as shown. By CPU 201 and multiple devices Device (Device0 206 ~ Device3 209) constitute a heterogeneous computing system 700a, Host CPU 201 is connected with system memory 203 through RC 202, multiple computing intensive devices Device0 206, Device2 207, Device1 208, Device3 209 carry out Host (host) and Deivce (device) interconnection.

[0115] In the data path of Host CPU 201, each Device (device) complies with PCIe protocol, and exists in PCIe topology as each Endpoint (EP) 210, receives various data types between transmission and Host CPU 201, such as Memory, Configure, Message, IO and other data types.

[0116] For the data transmission path between Device and Device, the present application improves the multi-device Device interconnection structure by introducing independent data path between Device and Device, such as by adding Device0 and Device2 data path 211, Device0 and Device1 data path 214, Device0 and Device3 data path 216.

[0117] The difference between the data paths of any two connected Devices and Host CPU based on such interconnection structure is that there is an independent data transmission path, separating Host CPU data and other Device data. In the computing intensive operation, such as a certain deep learning training process, Device0 206 ~ Device1 208 need to share a large amount of data transmission, and the present application can realize the interconnection between Device0 206 ~ Device1 208, and through the separated data path (such as data path 211 ~ 216), it can provide more efficient, low delay data transmission read-write mechanism, so as to ensure the real-time transmission characteristics of data, and further enhance the processing speed of high-performance computing coprocessor.

[0118] Figure 7bis another embodiment of the present application provides the structure diagram of the compatible PCIe structure of the interconnection system between devices, as shown in the figure. In the heterogeneous computing system 700b of the complete multi-core CPU (such as CPU core0 301a, CPU core1 301b, CPU core2 301c, CPU core3 301d) and multi-device (such as Device0 306a, Device1 306c, Device2 306b, Device3 306d), the plurality of CPU cores 301a, CPU core1 301b, CPU core2 301c, CPU core3 301d in the Host CPU respectively access the system memory 305 through the Host Memory Controller (HMC for short) 303 in the RC 302.

[0119] The plurality of computing-intensive Device0 306a, Device1 306c, Device2 306b, Device3 306d can directly exchange data between CPU and Device through external hub control bus (318a-318c) or indirectly through Switch (304a, 304b).

[0120] The external bus control unit in each Device (Device0 306a, Device1 306c, Device2 306b, Device3 306d) is EP type, that is, EP 310a, EP 310b, EP 310c, EP 310d, which is controlled by the RC 302 in the Host CPU. The Host RC 302 completes a series of control tasks such as PCIe link establishment, EP configuration enumeration, interrupt vector allocation reception and power consumption management.

[0121] After the initialization of the external hub control bus 318 (318a, 318b, 318c, 318d), data exchange between the Host and the Device is conducted via the bus. The Host CPU 301 (CPU core 301a, CPU core 1 301b, CPU core 2 301c, CPU core 3 301d) can read and write data in the storage unit 312 (312a, 312b, 312c, 312d) of the Device 306 (Device 0 306a, Device 1 306c, Device 2 306b, Device 3 306d), or the register of a client in the Device, such as the memory controller 311a. The Device can read and write data in the system memory 305 via the external hub control bus 318 (318a, 318b, 318c, 318d).

[0122] The external hub control bus 318 (318a, 318b, 318c, 318d) is a dedicated bus between the Host and the Device (Device 0 306a, Device 1 306c, Device 2 306b, Device 3 306d), and all data exchange between the Host and the Device is conducted via the bus. The dedicated external hub control bus 318 avoids the mixing of data paths between the Device and the Device, and between the Device and the Host, and the mutual occupation of bandwidth, thereby reducing the performance defects of the heterogeneous computing system.

[0123] The external hub bus control unit EP 310 (such as EP 310a, EP 310b, EP 310c, EP 310d) in the Device 306 (such as Device 0 306a, Device 1 306c, Device 2 306b, Device 3 306d) is controlled by the RC 302 in the Host. The EP 310 (such as EP 310a, EP 310b, EP 310c, EP 310d) receives and translates the Host CPU data register read-write control instructions to the internal high-speed on-chip bus 316 (such as 316a, 316b, 316c, 316d) of the Device. The hierarchical high-speed on-chip bus routes the translated Host CPU read-write commands to the memory controller 311 (such as 311a, 311b, 311c, 311d) in the Device. The memory controller 311 reads and writes the on-chip storage unit 312, and feeds back to the Host end.

[0124] The peer-to-peer interconnection control unit EP (313 / 314 / 315) in the device 306 (such as Device0 306a, Device1 306c, Device2 306b, Device3 306d) establishes PCIe link and reads and writes data with the peer control unit through an auto negotiation mechanism. The peer-to-peer interconnection bus realizes a physical data path between two connected devices, and data exchange between any connected devices can be performed through the bus. For example, the data read and write control command of the Device 306a can reach the peer Device 306d through the peer-to-peer interconnection bus data path 317e. The EP 314d receives and translates the data read and write control instruction from the Device 306a, and routes it to the storage control unit 311d through the internal high-speed on-chip bus 319d. The storage control unit 311d reads and writes the on-chip storage unit 312d according to the translated read and write control instruction, and returns the data to the Device 306a.

[0125] Therefore, based on the method for interconnecting devices compatible with the PCIe structure according to the application, two-by-two interconnection between devices (such as Device0 306a, Device1 306c, Device2 306b, Device3 306d) can be realized, and based on the interconnection, a PCIe link connection is established, so that the devices can read and write data, the data transmission delay of multi-device interconnection in a heterogeneous computing system can be reduced, the data bandwidth of multi-device interconnection can be expanded, and the bottleneck of the heterogeneous computing system can be further eliminated.

[0126] Figure 8 The flowchart of the method for interconnecting devices compatible with the PCIe structure according to another embodiment of the application is shown in the figure. The method for adaptively establishing peer-to-peer connection between multiple devices (Devices) according to the application includes the following steps:

[0127] Step 801: The Host CPU and each Device perform Host path link initialization, including performing physical link initialization.

[0128] Step 802: The Host and each Device perform higher-layer data transmission layer link connection and complete the enumeration process, thereby opening the entire Host path link, and then the Host and each Device can perform data interaction.

[0129] Step 803: The CPU configures the registers of the peer-to-peer interconnection control unit controller in each Device through the Host path.

[0130] Step 804, each Device begins to preliminarily attempt to establish a physical layer link connection.

[0131] Step 805, each Device performs auto negotiation interconnection during link establishment.

[0132] Step 806, according to the result of auto negotiation interconnection, each device peer interconnection control unit automatically switches the device type.

[0133] Step 807, each Device re-completes PCIe link establishment connection according to the switched device type.

[0134] Step 808, the adaptive peer interconnection link connection is completed, and each Device can send and receive data through the link.

[0135] A PCIe structure compatible multi-device interconnection system provided by the application is described below, and the PCIe structure compatible multi-device interconnection system described below can be correspondingly referred to the PCIe structure compatible multi-device interconnection method described above.

[0136] A PCIe structure compatible device interconnection system provided by the application includes a CPU, a root complex RC, a memory, a switch and a plurality of devices, the CPU is connected with the memory, the switch and the plurality of devices through the root complex, the device includes a first device and a second device,

[0137] The first device sends a training character containing a first preset link code to the second device; the second device switches the type of the data path that first receives the continuous training character containing the first preset link code; the second device sends a training character containing a second preset link code to the first device through the data path after type switching, so as to realize the interconnection of the first device and the second device.

[0138] Wherein, the first preset link code is used to define the type of the first device, the second preset link code is used to define the type of the second device, and the type of the data path is a root complex RC type or an endpoint device EP type.

[0139] The application further provides a PCIe structure compatible device interconnection chip, which is applied to a heterogeneous computing system, and the interconnection chip includes a first chip and a second chip, wherein:

[0140] The first chip sends a training character containing a PAD link number to the second chip;

[0141] The second chip switches the type of the data path that first receives the continuous training character containing the PAD link number;

[0142] The second chip sends the training character containing the non-PAD link number to the first chip through the data path after the type switching, so as to realize the inter-chip interconnection;

[0143] Wherein, the PAD link number is used to define the type of the first chip, and the non-PAD link number is used to define the type of the second chip; the first chip and the second chip both contain a plurality of data paths, and the type of the data path is RC type or EP type.

[0144] Optionally, the first chip and the second chip can be GPU chips.

[0145] Figure 9 An example of an entity structure diagram of an electronic device is shown in the figure, which can include a processor 810, a communications interface 820, a memory 830 and a communications bus 840, wherein the processor 810, the communications interface 820 and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the steps of the multi-device interconnection method compatible with the PCIe structure. Figure 9

[0146] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0147] ​In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the steps of the method for interconnecting multiple devices in a PCIe-compatible structure as described above.

[0148] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for interconnecting multiple devices in a PCIe-compatible structure as described above.

[0149] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., may be located in one place, or may be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0150] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0151] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A device interconnection method compatible with PCIe architecture, applied to a heterogeneous system, the heterogeneous system including a host and multiple devices connected to the host, characterized in that, The interconnection method includes: The first device changes the type of its own data path based on the preset control characters received from the second device or the host. The first device, after its type change, is interconnected with the second device; The preset control character is an instruction that conforms to the PCIe communication protocol and can control the first device to change the type of its own data path. The type of the data path is either root complex RC type or endpoint device EP type. The first device changes the type of its data path based on a preset control character received from the second device, including: The second device sends training characters containing the second preset link encoding to the first device; The first device switches the type of the data path that first receives consecutive training characters containing the second preset link encoding; The first device sends training characters containing a first preset link code to the second device through the data path after type switching, so as to realize the interconnection between the first device and the second device; The first preset link code is used to define the type of the data path of the first device, and the second preset link code is used to define the type of the data path of the second device. The type of the data path is either root complex RC type or endpoint device EP type.

2. The method according to claim 1, characterized in that, After the first device sends the training characters containing the first preset link encoding to the second device through the data path after type switching, the process includes: When the remaining data path of the first device receives a series of training characters containing the second preset link code, the type of the remaining data path of the first device is switched according to the type of the data path.

3. The method according to claim 1, characterized in that, Before the second device sends the training characters containing the second preset link encoding to the first device, it includes: A timeout mechanism is set for the first device and the second device to ensure that all data paths of the first device re-establish the PCIe link after type switching.

4. The method according to claim 3, characterized in that, The timeout mechanism operates through the following steps: The host confirms whether the peer-to-peer interconnection link between devices is successful by detecting the link status register in the device; If the link status remains False for a preset time, the peer-to-peer interconnection between the devices is considered unsuccessful. Reset the timing value and retry establishing the peer-to-peer connection between the devices; The host monitors the location and status information of each device by using the preset bus function device code, and changes or monitors the link status register between each device in real time.

5. The method according to any one of claims 1 to 4, characterized in that, In a configuration where the data path of the first device is of type RC and the data path of the second device is of type RC, the interconnection method includes: The second device sends training characters containing the second preset link encoding to the first device; The first device switches the type of the data path that first receives consecutive training characters containing the second preset link code from RC type to EP type; The first device sends training characters containing a first preset link code to the second device through the data path after type switching, so as to realize the interconnection between the second device and the first device.

6. The method according to any one of claims 1 to 4, characterized in that, In a configuration where the data path of the first device is of type EP and the data path of the second device is of type EP, the interconnection method includes: The second device sends training characters containing the second preset link encoding to the first device; The first device switches the type of the data path that first receives consecutive training characters containing the second preset link code from EP type to RC type; The first device sends training characters containing a first preset link code to the second device through the data path after type switching, so as to realize the interconnection between the first device and the second device.

7. The method according to claim 2, characterized in that, Under a data path structure where the first device's data path is of type RC and the second device's data path is of type EP, or under a data path structure where the first device's data path is of type EP and the second device's data path is of type RC, re-establishing the PCIe link includes: The second device sends training characters containing the second link encoding to the first device through the sending end of the data path; After receiving the training characters, the first device identifies the type of the second device based on the second link encoding and then configures its own first link encoding. The first device sends training characters containing the first link code to the second device through the sending end of the data path, so as to inform the second device of the first link code; After the second device receives the training characters and confirms the first link encoding, it completes the establishment of the PCIe link between the first device and the second device.

8. The method according to any one of claims 1 to 4, characterized in that, The first device changes the type of its data path based on a preset control character received from the host, including: The host assigns a bus function device code to each device to determine the coordinate position of each device in the entire mechanism system; Based on the preset bus function device code, the host can control the switching of the type of peer-to-peer interconnected devices.

9. A PCIe-compatible inter-device interconnect system, comprising a CPU, a root complex RC, a memory, a switch, and multiple devices, wherein the CPU is connected to the memory, the switch, and the multiple devices via the root complex, and the devices include a first device and a second device, characterized in that, The first device changes the type of its own data path according to the preset control character received from the second device or the CPU; the first device after changing the type is interconnected with the second device; The preset control character is an instruction that conforms to the PCIe communication protocol and can control the first device to change the type of its own data path. The type of the data path is either root complex RC type or endpoint device EP type. The first device changes the type of its data path based on a preset control character received from the second device, including: The second device sends training characters containing a second preset link code to the first device; the first device switches the type of the data path that first receives consecutive training characters containing the second preset link code; the first device sends training characters containing a first preset link code to the second device through the data path after the type switch, so as to realize the interconnection between the first device and the second device; The first preset link code is used to define the type of the first device, the second preset link code is used to define the type of the second device, and the data path type is either root composite RC type or endpoint device EP type.

10. A device interconnect chip compatible with PCIe architecture, applied in heterogeneous computing systems, characterized in that, The interconnect chip includes a first chip and a second chip, wherein: The first chip changes the type of its own data path according to the preset control character sent by the second chip; the first chip and the second chip are interconnected after the type is changed; The preset control character is an instruction that conforms to the PCIe communication protocol and can control the first chip to change the type of its own data path. The type of the data path is either root complex RC type or endpoint device EP type. The first chip changes the type of its data path based on a preset control character received from the second chip, including: The second chip sends training characters containing a second preset link encoding to the first chip; The first chip switches the type of the data path that first receives consecutive training characters containing the second preset link encoding; The first chip sends training characters containing a first preset link code to the second chip through the data path after type switching, so as to realize the interconnection between the first chip and the second chip; The first preset link code is used to define the type of the data path of the first chip, and the second preset link code is used to define the type of the data path of the second chip. The type of the data path is either root complex RC type or endpoint device EP type.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the device interconnection method for a PCIe-compatible architecture as described in any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the device interconnection method for a PCIe-compatible architecture as described in any one of claims 1 to 8.

Images