Multiprocessor system and method for configuring the same

Abstract

The present application provides a multi-processor system and a method for configuring a multi-processor system, and belongs to the field of computers. The present application realizes the connection between two domains by improving the existing PCIe-based system architecture. Since the PCIe device of the external domain will not be discovered by the processor of the local domain when the processor of the local domain is enumerated, the access conflict problem triggered by the discovery of the PCIe device of the external domain by the processor of the local domain is avoided. Therefore, when the PCIe switches of the two domains are interconnected, the processor or PCIe device of the local domain and other devices can still access the PCIe device of the external domain normally, thereby improving the communication reliability of cross-domain access. In addition, since the communication path of cross-domain access is established through the bus and the PCIe switches of the two domains, and does not rely on the forwarding of the processor, it helps to shorten the communication path of cross-domain access, thereby reducing the delay of cross-domain access while avoiding access conflicts.

Description

[0001] This application claims priority to Chinese patent application No. 202010660272.X filed on July 10, 2020, with invention name “A method and device for configuring a symmetrical load”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a multi-processor system and a method for configuring the multi-processor system. Background Art

[0003] Peripheral Component Interconnect Express (PCIe) is a high-speed serial computer expansion bus standard. PCIe has many advantages, such as higher maximum system bus throughput, smaller physical footprint, better bus device performance scalability, and more detailed error detection and reporting mechanisms. Therefore, PCIe has become a popular system architecture in modern computers.

[0004] However, in the current PCIe-based system architecture, such as the H2D system architecture, the PCIe switches of each PCIe domain are disconnected from each other, and the PCIe switches of each domain do not communicate. The communication relationship between each domain is established through the communication between the processors of different domains. However, this method has the problem that when the processor or PCIe device of the domain needs to access the PCIe device of other domains, the communication path is too long, resulting in a large delay in cross-domain access. However, if the communication relationship between the two domains is established directly by connecting the PCIe switches of the two PCIe domains, it will cause access conflicts. Summary of the invention

[0005] The embodiment of the present application provides a multi-processor system and a method for configuring the multi-processor system, which helps to reduce the delay of cross-domain access. The technical solution is as follows:

[0006] In the first aspect, a multi-processor system is provided, which includes at least two PCIe domains. Taking the case where the system includes a first PCIe domain and a second PCIe domain as an example, the first PCIe domain includes a first processor, a first PCIe switch connected to the first processor, and a first PCIe device connected to the first PCIe switch; the second PCIe domain includes a second processor, a second PCIe switch connected to the second processor, and a second PCIe device connected to the second PCIe switch, and the first PCIe switch and the second PCIe switch communicate through a bus; when the first processor performs enumeration, the second PCIe device in the second PCIe domain connected to the second PCIe switch will not be discovered by the first processor; the first processor or the first PCIe device accesses the second PCIe device through the bus.

[0007] In the multi-processor system provided in the first aspect, the system improves the existing PCIe-based system architecture and realizes the connection between two PCIe domains while avoiding access conflicts. Since the PCIe devices in the external domain will not be discovered by the processor in the local domain when the processor in the local domain is enumerated, the access conflict problem triggered by the discovery of the PCIe devices in the external domain by the processor in the local domain is avoided. Therefore, when the PCIe switches in the two domains are interconnected, the processors or PCIe devices in the local domain and other devices can still access the PCIe devices in the external domain normally, thereby improving the communication reliability of cross-domain access. In addition, since the communication path for cross-domain access is established through the bus and the PCIe switches in the two domains, and does not rely on the forwarding of the processor, it helps to shorten the communication path for cross-domain access, thereby reducing the delay of cross-domain access while avoiding access conflicts.

[0008] Optionally, when the second processor performs enumeration, the first PCIe device in the first PCIe domain connected to the first PCIe switch will not be discovered by the second processor; the second processor or the second PCIe device accesses the first PCIe device through the bus.

[0009] Optionally, the first PCIe switch includes a first interconnection port, the first interconnection port is used to communicate with the second PCIe switch, and the first interconnection port is configured to reject a scan instruction of the first processor when the first processor is enumerated.

[0010] Since the interconnect port rejects the scanning instruction of the central processor during the PCIe enumeration process, the interconnect port and the device connected to the interconnect port are hidden from the central processor, thereby providing a low-complexity and high-practicality implementation method for the feature that "PCIe devices in the external domain will not be discovered by the processor in this domain".

[0011] Optionally, the second PCIe switch includes a second interconnect port, where the second interconnect port is used to communicate with the first PCIe switch, and the second interconnect port is configured to reject a scan instruction of the second processor when the second processor is enumerated.

[0012] Optionally, the second PCIe switch includes a second interconnect port, where the second interconnect port is used to communicate with the first PCIe switch, and the second interconnect port is configured to reject a scan instruction of the first processor when the first processor is enumerated.

[0013] Similar to the solution of configuring the first interconnect port to reject the scanning instruction of the first processor, configuring the second interconnect port to reject the scanning instruction of the first processor can also achieve the effect that the device in the second PCIe domain will not be discovered by the first processor.

[0014] Optionally, the first PCIe switch includes a first interconnection port, the first interconnection port is used to communicate with the second PCIe switch, and the first interconnection port is configured to reject a scan instruction of the second processor when the second processor is enumerated.

[0015] Optionally, the first PCIe switch further includes a first host port, and the first PCIe switch is connected to the first processor through the first host port, wherein the first host port is not directly connected to the first interconnection port.

[0016] Since the host port and the interconnection port are not directly connected, the processor of the local domain is prevented from discovering the PCIe device of the other domain through the hardware connection relationship. Therefore, this implementation method helps to make "the PCIe device of the other domain will not be discovered by the processor of the local domain" more reliable, further reducing the probability of access conflicts.

[0017] Optionally, the second PCIe switch further includes a second host port, and the second PCIe switch is connected to the second processor through the second host port, wherein the second host port is not directly connected to the second interconnect port.

[0018] Optionally, the first PCIe switch includes a first firmware, and the first firmware is used to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

[0019] By solidifying the method of rejecting scan instructions in the PCIe switch, the implementation complexity is reduced and the practicality is high.

[0020] Optionally, the second PCIe switch includes second firmware, and the second firmware is used to configure the second interconnect port to reject the scan instruction of the second processor when the second processor is enumerated.

[0021] Optionally, the system further includes a basic input output system (BIOS), and the BIOS is used to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

[0022] Optionally, the system further includes a BIOS, configured to configure the second interconnect port to reject the scan instruction of the second processor when the second processor is enumerated.

[0023] Optionally, the system further includes a baseboard management controller (BMC); the BMC is used to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

[0024] Optionally, the system further includes a BMC; the BMC is used to configure the second interconnect port to reject the scan instruction of the second processor when the second processor is enumerated.

[0025] Optionally, the second PCIe switch includes second firmware, and the second firmware is used to configure the second interconnect port to reject a scan instruction of the first processor when the first processor is enumerated.

[0026] Optionally, the system further includes a BIOS, configured to configure the second interconnect port to reject a scan instruction of the first processor when the first processor is enumerated.

[0027] Optionally, the system further includes a BMC, and the BMC configures the second interconnect port to reject a scan instruction of the first processor when the first processor is enumerated.

[0028] Optionally, the second PCIe switch includes a second interconnection port, which is used to communicate with the first PCIe switch; the first interconnection port is configured with first cross-domain routing information, and the first processor or the first PCIe device accesses the second PCIe device through the first cross-domain routing information, and the first cross-domain routing information indicates the correspondence between the address information of the second PCIe device and the port identifier of the second interconnection port.

[0029] In the above manner, since the PCIe switch in the local domain is configured with the cross-domain routing information corresponding to the external domain, the access request can be forwarded through the cross-domain routing information, which helps to open up the cross-domain P2P channel.

[0030] Optionally, the first PCIe switch includes a first interconnection port, and the first interconnection port is used to communicate with the second PCIe switch; the second interconnection port is configured with second cross-domain routing information, and the second processor or the second PCIe device accesses the first PCIe device through the second cross-domain routing information, and the second cross-domain routing information indicates the correspondence between the address information of the first PCIe device and the port identifier of the first interconnection port.

[0031] Optionally, the first cross-domain routing information includes: a correspondence between a domain identifier of the second PCIe domain and a port identifier of the second interconnect port, and a correspondence between address information of the second PCIe device and a domain identifier of the second PCIe domain.

[0032] Optionally, the second cross-domain routing information includes: a correspondence between a domain identifier of the first PCIe domain and a port identifier of the first interconnect port, and a correspondence between address information of the first PCIe device and a domain identifier of the first PCIe domain.

[0033] Optionally, the first PCIe switch includes a first downstream port, which is connected to the first PCIe device; first local domain routing information is configured in the first interconnection port, and the first local domain routing information includes a correspondence between address information of the first PCIe device and a port identifier of the first downstream port.

[0034] In the above manner, since the PCIe switch in this domain is configured with the routing information of this domain, the access request sent from the external domain can reach the correct downstream port through the routing information of this domain, and then be transmitted to the PCIe device in this domain through the downstream port, thus helping to open up the cross-domain P2P channel.

[0035] Optionally, the second PCIe switch includes a second downstream port, which is connected to the second PCIe device; second local domain routing information is configured in the second interconnection port, and the second local domain routing information includes a correspondence between address information of the second PCIe device and a port identifier of the second downstream port.

[0036] Optionally, the first PCIe switch includes a first downstream port, in which first downstream port routing information is configured, and the first downstream port routing information indicates a correspondence between address information of the second PCIe device and a port identifier of the first interconnect port.

[0037] Through the above method, since the PCIe switch of this domain is configured with downstream port routing information, when the PCIe device of this domain needs to access the external domain, the access request of this domain can reach the correct interconnection port through the downstream port routing information, and then be transmitted to the external domain through the interconnection port, thus helping to open up the cross-domain P2P channel.

[0038] Optionally, the second PCIe switch includes a second downstream port, in which second downstream port routing information is configured, and the second downstream port routing information indicates a correspondence between address information of the first PCIe device and a port identifier of the second interconnect port.

[0039] Optionally, the first downstream port routing information includes a relationship between a domain identifier of the second PCIe domain and a port identifier of the first interconnect port, and a corresponding relationship between address information of the second PCIe device and a domain identifier of the second PCIe domain.

[0040] Optionally, the second downstream port routing information includes a relationship between a domain identifier of the first PCIe domain and a port identifier of the second interconnect port, and a corresponding relationship between address information of the first PCIe device and a domain identifier of the first PCIe domain.

[0041] Optionally, the address information of the first PCIe device includes an address window or a bus identifier of the first PCIe device.

[0042] Optionally, the address information of the second PCIe device includes an address window or a bus identifier of the second PCIe device.

[0043] Optionally, the first PCIe switch includes a first processing unit, and the first cross-domain routing information is stored in the first processing unit.

[0044] Optionally, the second PCIe switch includes a second processing unit, and the second cross-domain routing information is stored in the second processing unit.

[0045] Optionally, the first PCIe switch includes a first processing unit, and the first local domain routing information is stored in the first processing unit.

[0046] Optionally, the second PCIe switch includes a second processing unit, and the second local domain routing information is stored in the second processing unit.

[0047] Optionally, the first PCIe switch includes a first processing unit, and the first downstream port routing information is stored in the first processing unit.

[0048] Optionally, the second PCIe switch includes a second processing unit, and the second downstream port routing information is stored in the second processing unit.

[0049] Optionally, the first processor is a first central processing unit (CPU), and the second processor is a second CPU.

[0050] Optionally, the first PCIe device is a first embedded neural-network processing unit (NPU), a first tensor processing unit (TPU) or a first graphics processing unit (GPU), and the second PCIe device is a second NPU, a second TPU or a second GPU.

[0051] Optionally, the first PCIe switch includes a first downstream port, and the first downstream port is connected to a first PCIe device in the first PCIe domain;

[0052] The first PCIe switch is configured to receive a first access request through the first downstream port, where a source PCIe device of the first access request is the first PCIe device, and a destination PCIe device of the first access request is a second PCIe device in the second PCIe domain;

[0053] The first PCIe switch is used to add the domain identifier of the second PCIe domain to the first access request;

[0054] The first PCIe switch is used to send the first access request including the domain identifier to the second interconnection port through the first interconnection port.

[0055] Optionally, the first downstream port is configured with a ninth entry, the ninth entry including a correspondence between a domain identifier of the second PCIe domain and a port identifier of the first interconnect port;

[0056] The first PCIe switch is used to determine the first interconnection port according to the first access request and the ninth entry.

[0057] Optionally, the first downstream port is configured with an eleventh entry, the eleventh entry including a correspondence between an address window of the second PCIe device and a domain identifier of the second PCIe domain;

[0058] The first PCIe switch is used to determine the domain identifier of the second PCIe domain according to the first access request and the eleventh entry, and the destination address in the first access request belongs to the address window of the second PCIe device.

[0059] Optionally, the first downstream port is configured with a thirteenth entry, and the thirteenth entry includes a correspondence between a bus identifier corresponding to the second PCIe device and a port identifier of the first interconnect port;

[0060] The first PCIe switch is used to determine the first interconnection port according to the first access request and the thirteenth entry, wherein the first access request includes a bus identifier corresponding to the second PCIe device.

[0061] Optionally, the first interconnect port is configured with a first entry, the first entry including a correspondence between a domain identifier of the second PCIe domain and a port identifier of the second interconnect port;

[0062] The first PCIe switch is used to determine the second interconnection port according to the first access request and the first entry.

[0063] Optionally, the first PCIe switch includes a first downstream port, and the first downstream port is connected to a first PCIe device in the first PCIe domain;

[0064] The first PCIe switch is configured to receive a second access request through the first interconnection port, where the source PCIe device of the second access request is the second PCIe device in the second PCIe domain, and the destination PCIe device of the second access request is the first PCIe device;

[0065] The first PCIe switch is used to send the second access request to the first PCIe device through the first downstream port.

[0066] Optionally, the first PCIe switch includes a first downstream port, and the first downstream port is connected to a first PCIe device in the first PCIe domain;

[0067] The first interconnect port is configured with a fifth entry, the fifth entry including a correspondence between an address window of the first PCIe device and a port identifier of the first downstream port;

[0068] The second PCIe switch comprises a second downstream port, wherein the second downstream port is connected to a second PCIe device in the second PCIe domain;

[0069] The first PCIe switch is used to determine the first downstream port according to the second access request and the fifth entry, and the destination address in the second access request belongs to the address window of the first PCIe device.

[0070] Optionally, the first interconnect port is configured with a seventh entry, and the seventh entry includes a correspondence between a bus identifier corresponding to the first PCIe device and a port identifier of the first downstream port;

[0071] The first PCIe switch is used to determine the first downstream port according to the second access request and the seventh entry, wherein the second access request includes a bus identifier corresponding to the first PCIe device.

[0072] Optionally, the first downstream port is configured with a fifteenth entry, and the fifteenth entry includes a correspondence between an address window of the first PCIe device and an identifier of the first PCIe device;

[0073] The first PCIe switch is used to determine the first PCIe device according to the second access request and the fifteenth entry, and the destination address in the second access request belongs to the address window of the first PCIe device.

[0074] Optionally, the first downstream port is configured with a seventeenth entry, and the seventeenth entry includes a correspondence between a bus identifier corresponding to the first PCIe device and an identifier of the first PCIe device;

[0075] The first PCIe switch is used to determine the first PCIe device according to the second access request and the seventeenth entry, wherein the second access request includes a bus identifier corresponding to the first PCIe device.

[0076] In a second aspect, a method for applying to a multi-processor system is provided. The method includes: providing a first PCIe domain and a second PCIe domain, wherein the first PCIe domain includes a first processor, a first PCIe switch connected to the first processor, and a first PCIe device connected to the first PCIe switch and the first PCIe switch, and the second PCIe domain includes a second processor, a second PCIe switch connected to the second processor, and a second PCIe device connected to the second PCIe switch, and the first PCIe switch and the second PCIe switch communicate through a bus; configuring the first PCIe domain and the second PCIe domain so that when the first processor is enumerated, the second PCIe device in the second PCIe domain connected to the second PCIe switch will not be discovered by the first processor, wherein the first processor or the first PCIe device accesses the second PCIe device through the bus.

[0077] The specific details of the method provided in the second aspect can be found in the above-mentioned first aspect or any optional method of the first aspect, and will not be repeated here.

[0078] According to a third aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and when the at least one instruction is executed on the computer device, the computer device executes the method provided in the second aspect.

[0079] In a fourth aspect, a device for configuring a multi-processor system is provided, the device being used to configure the multi-processor system in the first aspect or any optional method of the first aspect. The device for configuring a multi-processor system includes at least one unit. In some embodiments, the unit in the device provided in the fourth aspect is implemented by software, and the unit in the device provided in the fourth aspect is a program module. In other embodiments, the unit in the device provided in the fourth aspect is implemented by hardware or firmware. The specific details of the device provided in the fourth aspect can be found in the first aspect or any optional method of the first aspect, and will not be repeated here.

[0080] In a fifth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium. When the at least one instruction is executed on a computer device, the computer device executes the method provided in the second aspect.

[0081] In a sixth aspect, a computer program product is provided, the computer program product comprising computer instructions stored in a computer-readable storage medium. When the computer program product is executed on a computer device, the computer device is caused to execute the method provided in the second aspect.

[0082] In a seventh aspect, a computer device is provided, the computer device comprising the multi-processor system provided by the first aspect or any optional manner of the first aspect, such as a server, a host or a personal computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a schematic diagram of a system architecture that adopts a P2P balanced connection method;

[0084] Figure 2 It is a schematic diagram of the system architecture using the H2D balanced connection method;

[0085] Figure 3 It is a schematic diagram of a system architecture using a dedicated bus;

[0086] Figure 4 is a schematic diagram of the architecture of a multi-processor system provided in an embodiment of the present application;

[0087] Figure 5 is a schematic diagram of the architecture of a multi-processor system provided in an embodiment of the present application;

[0088] Figure 6 is a schematic diagram of the architecture of a multi-processor system provided in an embodiment of the present application;

[0089] Figure 7 It is a structural diagram of a PCIe switch provided in an embodiment of the present application;

[0090] Figure 8 is a schematic diagram of software for configuring a multi-processor system provided by an embodiment of the present application;

[0091] Fig. 9 is a flow chart of a method for configuring a multi-processor system provided by an embodiment of the present application;

[0092] Fig.10 It is a schematic diagram of blocking communication access between a host port and an interconnection port provided in an embodiment of the present application.

[0093] Reference numerals:

[0094] 100-A multiprocessor system

[0095] 101-First PCIe domain

[0096] 102-Second PCIe domain

[0097] 110-First CPU

[0098] 111-Second CPU

[0099] 120-First PCIe switch

[0100] 1201-First host port

[0101] 1202-First interconnection port

[0102] 1203-First downstream port

[0103] 121-Second PCIe switch

[0104] 1211-Second host port

[0105] 1212-Second interconnection port

[0106] 1213-Second downstream port

[0107] 130-First PCIe device

[0108] 131-Second PCIe device

[0109] 200-A multiprocessor system

[0110] 300-A multiprocessor system

[0111] 400-A software for configuring a multiprocessor system

[0112] 410-Interconnect Configurator

[0113] 420-Topology Generator

[0114] 430-Address Allocator DETAILED DESCRIPTION

[0115] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0116] The system and method provided in the embodiment of the present application can be used as a solution to achieve symmetrical load of the system, and can be applied to any scenario where H2D channels and P2P channels need to be constructed. Among them, load symmetry includes the two meanings of load balancing between different CPUs and load balancing between different serial computer expansion bus standards (peripheral component interconnect express, PCIe) devices. Optionally, the system and method provided in the embodiment of the present application is applied to the scenario of artificial intelligence (AI) training on a single machine with multiple cards. The following introduces the scenario of AI training on a single machine with multiple cards and some situations faced by this scenario in application.

[0117] Currently, artificial intelligence technology is in the ascendant, and universities, enterprises, and research institutions are using a large number of heterogeneous computing servers for AI training. Since the AI ​​training process requires large-volume parameter synchronization between various training devices, it is necessary to establish a high-speed interconnected P2P channel between various training devices; at the same time, during the training process, data copying and message synchronization between the server CPU and the AI ​​acceleration computing card are required, so a high-speed and stable H2D channel is required between the server CPU and the training device.

[0118] The training device is generally an AI training card, such as a graphics processing unit (GPU), an embedded neural network processing unit (NPU), and other AI acceleration cards suitable for AI computing. AI training cards are generally PCIe devices, and AI training cards are connected to the server CPU through the PCIe bus. Therefore, the H2D channel for AI training is generally implemented based on the PCIe bus. As for P2P communication, in the scenario of stand-alone training, the P2P channel is generally also implemented based on the PCIe bus. When using the PCIe bus for P2P high-speed access, the PCIe bus path between all training devices is required to be the shortest to ensure performance.

[0119] In some application scenarios, the H2D channel and the P2P channel are constructed by a P2P balanced connection method or an H2D balanced connection method.

[0120] Please see attached Figure 1 , attached Figure 1 This is an example of using the PCIe bus for H2D communication and P2P communication under the P2P balanced connection method. Among them, CPU2 and PCIe switch 2 are not connected. The P2P balanced connection method ensures that the access path between each training device (GPU or NPU) is as short as possible and the bandwidth is reliable, but it cannot guarantee that the CPU (attached Figure 1 The PCIe access performance between CPU2) and each training device.

[0121] Please see attached Figure 2 , attached Figure 2 This is an example of using the PCIe bus for H2D communication and P2P communication under the H2D balanced connection method. Among them, PCIe switch 1 and PCIe switch 2 are not connected. The H2D balanced connection method can be used in conjunction with the CPU core binding strategy to allow training services on each CPU to access the training device as much as possible (see Appendix Figure 2However, each AI training device needs to synchronize parameters across CPUs, which will make P2P communication performance unstable in most cases (such as heavy CPU load).

[0122] It can be seen that both the P2P balanced connection method and the H2D balanced connection method cannot reconcile the conflicting performance requirements of H2D communication and P2P communication, either resulting in a degradation of H2D performance (P2P balanced connection method) or affecting P2P performance (H2D balanced connection method).

[0123] In other application scenarios, a dedicated bus for P2P communication is used to manage P2P communication between training devices. Figure 3 , a dedicated bus interface corresponding to the dedicated bus is additionally developed on each training device, and the dedicated bus interface of each training device is connected to the dedicated bus, so that different training devices can perform P2P communication through the dedicated bus.

[0124] However, although the solution using a dedicated bus reconciles the conflict between H2D and P2P in terms of performance requirements, it requires the development of dedicated hardware servers and training equipment for the dedicated bus, which is costly. In addition, the solution using a dedicated bus has limited application scenarios and is not suitable for training equipment without a dedicated bus interface.

[0125] In view of this, some embodiments of the present application can achieve equal H2D communication performance of multiple CPUs in the server to the training device in a single-machine multi-card training scenario, and the P2P communication capabilities between multiple training devices are basically consistent, meeting the needs of the above application scenarios. In addition, in a single-machine multi-card training scenario, high-speed P2P communication capabilities between training devices are achieved without relying on a new hardware bus or increasing hardware costs. The embodiments of the present application are described below from multiple perspectives such as hardware system architecture, software system architecture, and configuration method flow.

[0126] The following introduces the hardware system architecture provided by the embodiments of the present application.

[0127] The embodiment of the present application provides a multi-processor system. The multi-processor system includes multiple CPUs, multiple PCIe switches (PCIe switches) and multiple PCIe devices. For example, see the attached Figure 4 or attached Figure 5 , attached Figure 4 The system 100 shown is an example of a system architecture having 2 CPUs, 2 PCIe switches, and 8 PCIe devices. Figure 5 The system 200 shown is for an attached Figure 4 An illustration of system 100 is shown.

[0128] Optionally, the system provided in this embodiment, as shown in the attached Figure 4 The system 100 shown, Figure 5 The system architecture of the system 200 shown is a stand-alone system, and each CPU, each PCIe switch, and each PCIe device in the multi-processor system are integrated into the same hardware device. For example, the multi-processor system is integrated into the same server, the same host, or the same personal computer. Alternatively, different devices in the multi-processor system are distributed in different hardware devices.

[0129] The following is a detailed introduction to the multi-processor system through (1) to (16).

[0130] (1) PCIe domain

[0131] In the embodiments of the present application, the terms "PCIe domain", "PCIe network" and "PCIe system" are used interchangeably. A multi-processor system includes multiple PCIe domains. A PCIe domain includes a processor, a PCIe switch connected to the processor, and multiple PCIe devices connected to the PCIe switch. In some of the following embodiments, the processor is a CPU as an example for illustration. It should be noted that this embodiment does not limit the processor to a CPU. In other embodiments, the processor is other hardware other than the CPU. In other words, the executor of the actions such as enumeration and access to PCIe devices in the following embodiments may also be other hardware other than the CPU.

[0132] For example, see Appendix Figure 4 , attached Figure 4 The first PCIe domain 101 and the second PCIe domain 102 in the figure are examples of multiple PCIe domains. The first PCIe domain 101 includes a first central processor 110, a first PCIe switch 120 connected to the first central processor 110, and multiple PCIe devices connected to the first PCIe switch 120. Among them, the multiple PCIe devices connected to the first PCIe switch 120 include a first PCIe device 130. The second PCIe domain 102 includes a second central processor 111, a second PCIe switch 121 connected to the second central processor 111, and multiple PCIe devices 131 connected to the second PCIe switch 121. Among them, the multiple PCIe devices 131 connected to the second PCIe switch 121 include a second PCIe device 131.

[0133] For example, see Appendix Figure 5, CPU1 is an example of the first central processor 110, PCIe switch 1 is an example of the first PCIe switch 120, NPU0, NPU1, NPU2 and NPU3 are examples of the PCIe device 130. The first PCIe device 130 is, for example, any one of NPU0, NPU1, NPU2 or NPU3. CPU2 is an example of the second central processor 111, PCIe switch 2 is an example of the second PCIe switch 121, and NPU4, NPU5, NPU6 and NPU7 are examples of the PCIe device 131. The second PCIe device 131 is, for example, any one of NPU4, NPU5, NPU6 and NPU7.

[0134] (2) Topology of multiprocessor systems

[0135] Each PCIe domain in a multiprocessor system has a tree-like topology. A PCIe domain with a tree-like topology is also called a PCIe tree or a PCIe topology tree. The root node in the PCIe topology tree is the CPU, and the leaf nodes in the PCIe topology tree are PCIe devices.

[0136] In some embodiments, the CPU is integrated with a root complex (RC); in other embodiments, the CPU is separately configured from the RC. The RC is used to connect the processor and the memory subsystem to one or more PCIe switches. The RC is used to generate access requests on behalf of the CPU.

[0137] Within each PCIe domain, each PCIe port and each PCIe device is assigned a unique bus number, PCIe device number, and function number. Figure 4 , attached Figure 4 The system 100 shown includes two PCIe topology trees (i.e., two PCIe domains). One PCIe topology tree has the first central processor 110 as a root node and the PCIe device 130 as a leaf node. The other PCIe topology tree has the second central processor 111 as a root node and the PCIe device 131 as a leaf node. Figure 5 , attached Figure 5 The system 200 shown includes two PCIe topology trees (i.e., 2 PCIe domains). One PCIe topology tree has CPU1 as the root node, and NPU0, NPU1, NPU2, and NPU3 as leaf nodes. The other PCIe topology tree has CPU2 as the root node, and NPU4, NPU5, NPU6, and NPU7 as leaf nodes.

[0138] (3) Domain ID

[0139] In some embodiments, PCIe domains in a multiprocessor system are identified by domain IDs. PCIe devices in the same PCIe domain have the same domain ID. PCIe devices in different PCIe domains have different domain IDs. For example, see the attached Figure 5 , attached Figure 5 The domain 1 in FIG. 1 is an example of the first PCIe domain 101, and the domain identifier of the domain 1 is, for example, 1 or domain1. Figure 5 The domain 2 in the figure is an example of the second PCIe domain 102, and the domain identifier of the domain 2 is, for example, 2 or domain2.

[0140] (4) PCIe switch

[0141] A PCIe switch is used to forward access requests from devices in different or the same PCIe domain. A PCIe switch includes multiple PCIe ports, and the functions and connection objects of different PCIe ports are different. Specifically, a PCIe switch includes a fabric port, a downstream port (DP), and a host port.

[0142] (5)Interconnection port

[0143] The interconnection port is used to communicate with other PCIe switches. The interconnection ports of different PCIe switches are connected. A PCIe switch includes one or more interconnection ports. In the case where the PCIe switch includes multiple interconnection ports, different interconnection ports are connected to different PCIe switches. For example, see the attached Figure 4 , the first PCIe switch 120 includes a first interconnection port 1202, the second PCIe switch 121 includes a second interconnection port 1212, and the second interconnection port 1212 is connected to the first interconnection port 1202 through a PCIe bus. The first interconnection port 1202 is used to communicate with the second PCIe switch 121. The second interconnection port 1212 is used to communicate with the first PCIe switch 120. For example, see the attached Figure 5 , PCIe switch 1 includes interconnection port a, PCIe switch 2 includes interconnection port b, interconnection port a and interconnection port b are connected via a PCIe bus. Wherein, interconnection port a is an example of the first interconnection port 1202, and interconnection port b is an example of the second interconnection port 1212.

[0144] (6) Downstream port

[0145] The downstream port is used to communicate with the PCIe device. The downstream port of the PCIe switch is connected to the PCIe device. A PCIe switch includes at least one downstream port, and each downstream port is connected to at least one PCIe device.

[0146] For example, see Appendix Figure 4 The first PCIe switch 120 includes a first downstream port 1203 connected to the first PCIe device 130 in the first PCIe domain 101. The second PCIe switch 121 includes a second downstream port 1213 connected to the second PCIe device 131 in the second PCIe domain 102.

[0147] For example, see Appendix Figure 5 , the downstream port is attached Figure 5 The four downstream ports are referred to as DP0 to 3. Among them, DP0 of PCIe switch 1 is connected to NPU0; DP1 of PCIe switch 1 is connected to NPU1; DP2 of PCIe switch 1 is connected to NPU2. DP3 of PCIe switch 1 is connected to NPU3. DP0 of PCIe switch 2 is connected to NPU4; DP1 of PCIe switch 2 is connected to NPU5; DP2 of PCIe switch 2 is connected to NPU6. DP3 of PCIe switch 2 is connected to NPU7. Among them, NPU0, NPU1, NPU2 or NPU3 is an example of the first PCIe device 130. NPU4, NPU5, NPU6 or NPU7 is an example of the second PCIe device 131.

[0148] (7)Host port

[0149] The host port is used to communicate with the central processing unit. The host port of the PCIe switch is connected to the central processing unit. For example, see the attached Figure 4 , the first PCIe switch 120 includes a first host port 1201, and the first host port 1201 is connected to the first central processor 110. The second PCIe switch 121 includes a second host port 1211, and the second host port 1211 is connected to the second central processor 111. For example, see the attached Figure 5 , the host port of PCIe switch 1 is connected to CPU 1. The host port of PCIe switch 2 is connected to CPU 2.

[0150] (8) Management port

[0151] In some embodiments, see Appendix Figure 7 The PCIe switch also includes a management port. The management port is connected to the central processing unit. The management port is used to receive configuration instructions from the central processing unit.

[0152] (9) Firmware

[0153] In some embodiments, see Appendix Figure 7The PCIe switch also includes firmware. The firmware includes program codes that implement the relevant functions of the PCIe switch. The PCIe switch executes the method corresponding to the PCIe switch in this embodiment by executing the program codes in the firmware.

[0154] (10) PCIe devices

[0155] In some embodiments, the PCIe device is a processor. For example, the PCIe device is an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The PCIe device is, for example, a single-core processor or a multi-core processor.

[0156] In some other embodiments, the PCIe device is a peripheral device. For example, the PCIe device is an output device or an input device. The output device communicates with the central processing unit and can display information. For example, the output device is a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device or a projector. The input device communicates with the central processing unit and can receive user input. For example, the input device can be a mouse, a keyboard, a touch screen device or a sensor device.

[0157] In some embodiments, the PCIe device is a training device. A training device refers to a device used to perform AI training tasks. For example, the PCIe device is an AI processor (also known as an AI training card). For example, the PCIe device is a GPU, NPU, tensor processing unit (TPU) or other chips customized for machine learning. Among them, the tasks of AI training include but are not limited to calculating at least one of gradient values ​​and calculating model parameters.

[0158] In other embodiments, the PCIe device is a computing device of other types other than the training device, for example, the PCIe device is any type of accelerator card or a dedicated processor.

[0159] In other embodiments, the PCIe device is a storage device, for example, a solid state drive (SSD), a hard disk drive (HDD), or a dual-inline memory module (DIMM).

[0160] In some other embodiments, the PCIe device is a network device. For example, the PCIe device is a network interface card (NIC).

[0161] (11) Connection relationship between CPU and PCIe switch

[0162] In some embodiments, the CPU and the PCIe switch in the same PCIe domain are connected via a PCIe bus. Figure 4 In the illustrated scenario, the first central processor 110 is connected to the first PCIe switch 120 via a PCIe bus, and the second central processor 111 is connected to the second PCIe switch 121 via a PCIe bus.

[0163] (12) Connection relationship between PCIe switch and PCIe device

[0164] In some embodiments, a PCIe switch in the same PCIe domain is connected to each PCIe device via a PCIe bus. Figure 4 In the illustrated scenario, the first PCIe switch 120 is connected to the first PCIe device 130 via a PCIe bus. The second PCIe switch 121 is connected to the second PCIe device 131 via a PCIe bus. Figure 5 , PCIe switch 1 is connected to four NPUs through four PCIe buses. Specifically, PCIe switch 1 is connected to NPU0 through bus 10, PCIe switch 1 is connected to NPU1 through bus 11, PCIe switch 1 is connected to NPU2 through bus 12, and PCIe switch 1 is connected to NPU3 through bus 13. PCIe switch 2 is connected to four NPUs through four PCIe buses. Specifically, PCIe switch 2 is connected to NPU4 through bus 80, PCIe switch 2 is connected to NPU5 through bus 81, PCIe switch 2 is connected to NPU6 through bus 82, and PCIe switch 2 is connected to NPU7 through bus 83.

[0165] (13) Connection relationship between different PCIe switches

[0166] In some embodiments, PCIe switches in different PCIe domains are connected via a bus (such as a PCIe bus), and PCIe switches in different PCIe domains communicate with each other via the bus. Therefore, a connection relationship is established between different PCIe domains via the PCIe switch and the bus, so as to communicate based on the connection relationship.

[0167] For example, in the attached Figure 4 In the illustrated scenario, the first PCIe switch 120 is connected to the second PCIe switch 121 via a PCIe bus, and the first PCIe switch 120 communicates with the second PCIe switch 121 via the PCIe bus and a PCIe-related communication protocol. Since the first PCIe switch 120 communicates with the second PCIe switch 121 via the PCIe bus, the first processor or the first PCIe device 130 accesses the second PCIe device 131 via the PCIe bus, and the second processor or the second PCIe device 131 accesses the first PCIe device 130 via the PCIe bus. For example, see the attached Figure 5 , PCIe switch 1 and PCIe switch 2 are connected through a PCIe bus.

[0168] It should be noted that the present embodiment does not limit the bus interconnecting different PCIe switches to be a bus of the PCIe bus type. In other embodiments, the bus interconnecting different PCIe switches includes not only a PCIe bus but also other types of buses. For example, the connection relationship between the first PCIe switch 120 and the second PCIe switch may also be: first PCIe switch 120-PCIe bus-other types of buses-PCIe bus-second PCIe switch 121.

[0169] Another point to be explained is that this embodiment does not limit the number of buses interconnecting different PCIe switches to only one. In other embodiments, the number of buses interconnecting different PCIe switches includes multiple buses.

[0170] Another point to be explained is that the PCIe bus is an example of a bus used to connect different PCIe switches. In other embodiments, different PCIe switches are not connected through the PCIe bus, but through other buses other than the PCIe bus. This embodiment does not limit the type of bus connecting different PCIe switches.

[0171] (14) Connection relationship between CPUs in different PCIe domains

[0172] In some embodiments, the central processors of different PCIe domains in a multi-processor system are connected via a CPU bus. Figure 4In the illustrated scenario, the first central processor 110 is connected to the second central processor 111 via a CPU bus. Figure 5 , CPU1 and CPU2 are connected through the CPU bus.

[0173] (15) Baseboard management controller (BMC)

[0174] In some embodiments, see Appendix Figure 7 , a multiprocessor system also includes a BMC. The BMC is connected to each PCIe switch in the system. For example, an out-of-band channel is established between the PCIe switch and the BMC; see Appendix Figure 7 In some embodiments, the out-of-band channel is established through a two-wire serial (Inter-Integrated Circuit, I2C) bus, and the PCIe switch manages the PCIe port connected to the BMC based on the I2C controller. In other embodiments, the out-of-band channel is established through a system management bus (system management bus, SMbus), and the PCIe switch manages the PCIe port connected to the BMC based on the SMbus controller.

[0175] (16) The relationship between the host port and the interconnection port.

[0176] In some embodiments, the host port is not directly connected to the interconnect port. Figure 4 , the first host port 1201 in the first PCIe switch 120 is not directly connected to the first interconnection port 1202 in the first PCIe switch 120. The second host port 1211 in the second PCIe switch 121 is not directly connected to the second interconnection port 1212 in the second PCIe switch 121.

[0177] Among them, indirect connection means that the host port and the interconnection port are not connected by a line. It should be noted that the term "indirect connection" does not exclude the case of indirect connection. In other words, the host port and the interconnection port may also be indirectly connected. For example, although the first host port 1201 and the first interconnection port 1202 are not directly connected by a line, the first host port 1201 is connected to other devices, and the other devices are connected to the first interconnection port 1202. This situation also falls within the scope of "the first host port and the first interconnection port are not directly connected".

[0178] Since the host port and the interconnection port are not directly connected, the processor of the local domain is prevented from discovering the PCIe device of the other domain through the hardware connection relationship. Therefore, this implementation method helps to make "the PCIe device of the other domain will not be discovered by the processor of the local domain" more reliable, and further reduces the probability of access conflicts. For example, since the first host port 1201 is not directly connected to the first interconnection port 1202, the first central processor 110 is prevented from discovering the second PCIe device 131 through the hardware connection relationship (such as the first host port 1201→the first interconnection port 1202→the second interconnection port 1212→the second PCIe device 131), thereby helping to make the feature of "the second PCIe device 131 will not be discovered by the first central processor 110" more reliable.

[0179] It is worth noting that this embodiment does not limit the number of CPUs in a multi-processor system. Figure 4 , Attachment Figure 5 The scenario of 2 CPUs shown is only for illustration. In other embodiments, the multiprocessor system has a greater number of CPUs than 2.

[0180] Another point worth mentioning is that this embodiment does not limit the number of PCIe devices in a multi-processor system. Figure 4 , Attachment Figure 5 The scenario of 8 PCIe devices shown is for illustration only. In other embodiments, the number of PCIe devices in the multiprocessor system is less than 8, such as the multiprocessor system has 4 PCIe devices. In other embodiments, the number of PCIe devices in the multiprocessor system is more than 8, such as the number of PCIe devices in the multiprocessor system is dozens or hundreds, or more.

[0181] The above introduces the hardware architecture of the multi-processor system. Figure 8 , introduces the software architecture of multiprocessor systems.

[0182] See attached Figure 8 , attached Figure 8 The software 400 shown is an example of the software architecture of the system 100 or system 200 introduced above. The software 400 includes an interconnection configurator 410, a topology generator 420, and an address distributor 430. The interconnection configurator 410, the topology generator 420, and the address distributor 430 are software modules. For example, the interconnection configurator 410, the topology generator 420, and the address distributor 430 are modules generated after the processor reads the program code stored in the memory. The software 400 is, for example, PCIe initialization configuration software.

[0183] The interconnection configurator 410 is used to send a mode configuration instruction to the PCIe switch, and the mode configuration instruction is used to configure the mode of the PCIe switch to the Fabric mode. In some embodiments, the interconnection configurator 410 runs in the BMC, and the mode configuration instruction is sent as the BMC is started. In other embodiments, if the interconnection relationship is fixed and does not need to be switched, the interconnection configurator 410 is solidified in the firmware configuration of the PCIe switch, and each time the system is started, the interconnection configurator 410 is automatically configured to the specified interconnection relationship.

[0184] The topology generator 420 is used to generate the topological relationship between the PCIe device and the CPU. For example, the topology generator 420 generates the topological relationship between the PCIe device and the CPU according to the PCIe scanning algorithm. In some embodiments, the topology generator 420 is used to assign domain identifiers. For example, the topology generator 420 assigns different domain IDs to all PCIe domains in the system. In some embodiments, the topology generator 420 runs in the BIOS.

[0185] The address mapper is used to configure the routing information of the PCIe port so that the PCIe port can route and forward the access request based on the routing information. For example, the address mapper generates routing information according to the address and ID of each PCIe device in each PCIe domain divided by the topology generator 420; the address mapper carries the generated routing information in the configuration instruction and sends it to the interconnection port and the downstream port, thereby configuring the routing information to the interconnection port and the downstream port. In some embodiments, the address mapper is software running on the CPU. For example, the address mapper runs in the BIOS.

[0186] This embodiment does not limit the software 400 to run entirely on the same hardware. In some embodiments, the modules in the software 400 run on different hardware respectively. For example, some modules run on the CPU, some modules run on the BMC, and some modules run on the PCIe switch. For example, the topology generator 420 and the address distributor 430 both run on the CPU, the interconnection configurator 410 is the firmware of the PCIe switch or the interconnection configurator 410 is the software running on the BMC. In other embodiments, all the software modules in the software 400 run on the CPU, or all run on the BMC, or all run on the PCIe switch.

[0187] The above introduces the hardware architecture and software architecture of the multi-processor system. The multi-processor system described above improves the existing PCIe-based system architecture to achieve connection between two domains. When the processor of this domain is enumerated, the PCIe device of the external domain will not be discovered by the processor of this domain. Therefore, the PCIe device of the external domain is hidden from the processor of this domain, avoiding the access conflict problem triggered by the PCIe device of the external domain being discovered by the processor of this domain. Therefore, when the PCIe switches of the two domains are interconnected, the processor or PCIe device of the local domain can still access the PCIe device of the external domain normally, which improves the communication reliability of cross-domain access. In addition, since the communication path of cross-domain access is established through the bus and the PCIe switches of the two domains, and does not rely on the forwarding of the processor, it helps to shorten the communication path of cross-domain access, thereby reducing the delay of cross-domain access while avoiding access conflicts.

[0188] For example, in the attached Figure 4 In the system shown, when the first processor performs enumeration, the second PCIe device 131 in the second PCIe domain connected to the second PCIe switch will not be discovered by the first processor; when the second processor performs enumeration, the first PCIe device 130 in the first PCIe domain connected to the first PCIe switch will not be discovered by the second processor. Then, since the processors of the two domains will not discover the PCIe devices in the other domain, the two domains will not make errors when enumerating when the PCIe switches are interconnected.

[0189] There are many specific implementation methods for the feature that "PCIe devices in the external domain will not be discovered by the processor in the local domain". In some of the following embodiments, this feature is implemented by "rejecting the scanning instructions of the processor" as an example.

[0190] It is worth noting that this embodiment does not limit which domain's device is configured to reject the processor's scan instruction to achieve "PCIe devices in the external domain will not be discovered by the processor in this domain". In some embodiments, the interconnection port in the PCIe switch in the local domain is configured to reject the scan instruction of the processor in the local domain during the enumeration process, thereby preventing the processor in the local domain from discovering PCIe devices in the external domain through the PCIe switch in the local domain. In other embodiments, the interconnection port in the PCIe switch in the external domain is configured to reject the scan instruction of the central processor in the local domain during the enumeration process, thereby preventing the processor in the local domain from discovering PCIe devices in the external domain through the PCIe switch in the external domain.

[0191] In other embodiments, other methods other than "rejecting the processor's scanning instructions" may be used to achieve "PCIe devices in the external domain will not be discovered by the processor in this domain". This embodiment does not limit the specific implementation method of the current "PCIe devices in the external domain will not be discovered by the processor in this domain".

[0192] The following describes how to configure the system architecture described above.

[0193] See attached Fig. 9 , the multi-processor system configuration method 500, for example, includes the following steps S501 to S504.

[0194] Step S501: configure a PCIe port of a PCIe switch as a host port, a downstream port or an interconnection port.

[0195] From the hardware perspective, step S501 is performed by, for example, a central processing unit, a BMC, or a PCIe switch itself. Figure 4 In the system shown, the first central processor 110 executes S501 for the first PCIe switch 120, and the second central processor 111 executes S501 for the second PCIe switch 121. For another example, the BMC executes S501 for the first PCIe switch 120 and the second PCIe switch 121, respectively. For another example, the first PCIe switch 120 executes S501 through the first firmware, and the second PCIe switch 121 executes S501 through the second firmware.

[0196] From the perspective of software, step S501 is performed by, for example, an interconnect configurator. For example, the interconnect configurator is first burned into the firmware of the PCIe switch, and then the interconnect configurator is run to perform step S501.

[0197] Taking the interconnection configurator executing S501 as an example, S501 specifically includes the following S5011 to S5013.

[0198] S5011. The interconnection configurator determines a PCIe port connected to the CPU among multiple PCIe ports of the PCIe switch, and configures the PCIe port connected to the CPU as a host port.

[0199] S5012: The interconnect configurator determines a PCIe port connected to the PCIe device from among multiple PCIe ports of the PCIe switch, and configures the PCIe port connected to the PCIe device as a downstream port.

[0200] For example, in the attached Figure 5In the scenario of dual CPUs and 8 NPUs shown, the interconnection configurator running on CPU1 or PCIe switch 1 determines the four PCIe ports on PCIe switch 1 connected to the four NPUs NPU0 to NPU3, and the interconnection configurator configures the four PCIe ports connected to the four NPUs as four downstream ports. Similarly, the interconnection configurator running on CPU2 or PCIe switch 2 determines the four PCIe ports on PCIe switch 2 connected to the four NPUs NPU4 to NPU7, and the interconnection configurator configures the four PCIe ports connected to the four NPUs as four downstream ports.

[0201] S5013: The interconnection configurator determines a PCIe port connected to other PCIe switches among multiple PCIe ports of the PCIe switch, and configures the PCIe port connected to other PCIe switches as an interconnection port.

[0202] For example, in the attached Figure 5 In the scenario of dual CPUs and eight NPUs shown in the figure, the interconnection configurator running on CPU1 or PCIe switch 1 determines the PCIe port on PCIe switch 1 connected to PCIe switch 2, and the interconnection configurator configures the PCIe port connected to PCIe switch 2 as interconnection port a. Similarly, the interconnection configurator running on CPU2 or PCIe switch 2 determines the PCIe port on PCIe switch 2 connected to PCIe switch 1, and the interconnection configurator configures the PCIe port connected to PCIe switch 1 as interconnection port b.

[0203] Step S502: During the power-on startup process, the basic input output system (BIOS) running on the CPU performs enumeration.

[0204] Enumeration refers to the process of discovering PCIe devices after a processor (such as a CPU) is started. The processor knows which PCIe devices are in the system through enumeration so as to allocate address space (such as bus number, address window, etc.) to each PCIe device. Enumeration is performed, for example, when the system is initialized.

[0205] The scan instruction is an instruction sent by the processor during enumeration. Specifically, during the enumeration process, the processor will send a scan instruction; after the PCIe device receives the scan instruction, the PCIe device will respond to the scan instruction, generate and send a reply message to the processor; after receiving the reply message, the processor allocates address space for the PCIe device.

[0206] During the enumeration process, the interconnected ports of the PCIe switch will reject the BIOS scan instructions to hide the CPU. The host port and downstream port respond normally to the BIOS scan instructions and configuration commands to make them visible to the CPU. Through enumeration, the tree topology of the PCIe will be scanned out. After the enumeration is completed, each CPU is an independent PCIe tree.

[0207] Step S503: The topology generator regards each PCIe tree as a PCIe domain and allocates a corresponding domain ID to each PCIe domain, wherein different domain IDs are allocated to different PCIe domains.

[0208] Step S504: The address mapper collects base address register (BAR) space addresses and bus address information of PCIe devices in each PCIe domain, and configures routing information for each PCIe port according to the collected addresses.

[0209] The above describes the overall process of the configuration method. The following provides examples of how to configure interconnection ports and how to configure routing information.

[0210] The interconnection port in the PCIe switch is configured to reject the scanning instruction of the central processor during the enumeration process. The downstream port and the host port in the PCIe switch are configured to respond to the scanning instruction of the central processor during the enumeration process.

[0211] For example, in the case where the interconnection port in the PCIe switch of the domain is configured to reject the scan instruction of the processor of the domain during the enumeration process, see the attached Figure 4 The first PCIe switch 120 includes a first interconnect port 1202, a first downstream port 1203 and a first host port 1201. The first interconnect port 1202 is configured to reject a scan instruction of the first central processor 110 when the first central processor 110 is enumerated, the first downstream port 1203 is configured to respond to a scan instruction of the first central processor 110 when the first central processor 110 is enumerated, and the first host port 1201 is configured to respond to a scan instruction of the first central processor 110 when the first central processor 110 is enumerated.

[0212] Similarly, the second PCIe switch 121 includes a second interconnect port 1212, a second downstream port 1213 and a second host port 1211. The second interconnect port 1212 is configured to reject the scanning instruction of the second central processor 111 when the second central processor 111 is enumerated, the second downstream port 1213 is configured to respond to the scanning instruction of the second central processor 111 when the second central processor 111 is enumerated, and the second host port 1211 is configured to respond to the scanning instruction of the second central processor 111 when the second central processor 111 is enumerated.

[0213] For another example, in the case where the interconnection port in the PCIe switch of the external domain is configured to reject the scan instruction of the processor of the local domain during the enumeration process, the second interconnection port 1212 is configured to reject the scan instruction of the first central processor 110 when the first central processor 110 is enumerated. The first interconnection port 1202 is configured to reject the scan instruction of the second central processor 111 when the second central processor 111 is enumerated.

[0214] There are multiple implementation methods for configuring the interconnection port to reject scanning instructions. The following uses implementation method 1, implementation method 2, and implementation method 3 as examples to illustrate.

[0215] Implementation method 1: Configure by BIOS.

[0216] When the first implementation is adopted, the BIOS is responsible for configuring the interconnection port in the PCIe switch to reject the scanning instruction of the first central processor 110 and / or the second central processor 111 during the enumeration process.

[0217] In some embodiments, the action of BIOS configuring the interconnect port is implemented by interacting with the management port of the PCIe switch. Figure 7 , the BIOS running on the CPU generates a configuration instruction and sends the configuration instruction to the management port, and the configuration instruction is used to instruct to reject the scanning instruction of the first central processor 110 and / or the second central processor 111 during the enumeration process. The PCIe switch receives the configuration instruction through the management port, and in response to the configuration instruction, configures the interconnection port to reject the scanning instruction of the first central processor 110 and / or the second central processor 111. In some embodiments, the action of configuring the interconnection port is specifically implemented by accessing the register of the interconnection port. For example, the management port sends the configuration instruction to the firmware in the PCIe switch, and the firmware in the PCIe switch writes the configuration instruction to the register of the interconnection port, and the register of the interconnection port saves the configuration instruction. During the enumeration process, the interconnection port reads the configuration instruction from the register and executes the configuration instruction, thereby rejecting the scanning instruction of the BIOS. For example, in the attached Figure 4In the system architecture shown, the BIOS is used to configure the first interconnect port 1202 and the second interconnect port 1212 to reject the scan instructions of the first central processor 110 and / or the second central processor 111 during the enumeration process.

[0218] It is worth noting that this embodiment does not limit which CPU the BIOS for configuring the interconnection port to reject the scan instruction runs on. Figure 4 Taking the system architecture shown as an example, configuring the interconnect port to reject the CPU of the scan instruction includes but is not limited to the following cases 1 to 4.

[0219] Case 1: BIOS includes multiple threads, and different threads of BIOS run on different central processing units. Optionally, in this case, different central processing units configure the first interconnection port 1202 and the second interconnection port 1212 to reject scan instructions during the enumeration process through different threads of BIOS. For example, BIOS includes a first thread and a second thread, the first thread runs on the first central processing unit 110, and the second thread runs on the second central processing unit 111. Optionally, in this case, the first central processing unit 110 configures the first interconnection port 1202 to reject scan instructions during the enumeration process through the first thread, and the second central processing unit 111 configures the second interconnection port 1212 to reject scan instructions during the enumeration process through the second thread.

[0220] Case 2: BIOS runs on the first central processor 110. Optionally, in this case, the first central processor 110 configures the first interconnection port 1202 and the second interconnection port 1212 to reject scanning instructions during the enumeration process through BIOS.

[0221] Case 3: BIOS runs on the second central processing unit 111. Optionally, in this case, the second central processing unit 111 configures the first interconnection port 1202 and the second interconnection port 1212 to reject scanning instructions during the enumeration process through BIOS.

[0222] Case 4: BIOS runs on other CPUs other than the first CPU 110 and the second CPU 111. Optionally, in this case, other CPUs configure the first interconnection port 1202 and the second interconnection port 1212 to reject scan instructions during enumeration through BIOS.

[0223] Implementation method 2: Configure by BMC.

[0224] For example, see Appendix Figure 7, the BMC generates a configuration instruction and sends the configuration instruction to the PCIe switch through an out-of-band channel; the PCIe switch receives the configuration instruction through the out-of-band channel, and in response to the configuration instruction of the BMC, configures the interconnection port to reject the scanning instruction. BMC is used to configure the first interconnection port 1202 and the second interconnection port 1212 to reject the scanning instruction during the enumeration process. Specifically, the BMC generates and sends a configuration instruction to the first PCIe switch 120, and the first PCIe switch 120 configures the first interconnection port 1202 to reject the scanning instruction of the first central processor 110 and / or the second central processor 111 in response to the configuration instruction of the BMC. The BMC generates and sends a configuration instruction to the second PCIe switch 121, and the second PCIe switch 121 configures the second interconnection port 1212 to reject the scanning instruction of the first central processor 110 and / or the second central processor 111 in response to the configuration instruction of the BMC.

[0225] Implementation method three: configuration through firmware.

[0226] For example, the program code for the interconnection port to reject the scan instruction during the enumeration process is burned into the firmware of the PCIe switch, so as to be fixed in the PCIe switch. Whenever the system starts, the interconnection port will reject the scan instruction based on the firmware configuration. For example, the first PCIe switch 120 includes a first firmware, and the first firmware is used to configure the first interconnection port 1202 to reject the scan instruction of the first central processor 110 and / or the second central processor 111 during the enumeration process; the second PCIe switch 121 includes a second firmware, and the second firmware is used to configure the second interconnection port 1212 to reject the scan instruction of the first central processor 110 and / or the second central processor 111 during the enumeration process.

[0227] Through the above configuration method, since the interconnection port rejects the scanning instruction, the interconnection port and the device connected to the interconnection port are hidden from the central processor; since the downstream port and the host port respond to the scanning instruction, the downstream port, the device connected to the downstream port, the host port, and the device connected to the host port are visible to the central processor. Figure 4In the illustrated scenario, the first interconnection port 1202 and the second PCIe switch 121 connected to the first interconnection port 1202 are hidden from the first central processor 110, the first downstream port 1203 and the first PCIe device 130 connected to the first downstream port 1203 are visible to the first central processor 110, and the first host port 1201 is visible to the first central processor 110. The second interconnection port 1212 and the first PCIe switch 120 connected to the second interconnection port 1212 are hidden from the second central processor 111, the second downstream port 1213 and the second PCIe device 131 connected to the second downstream port 1213 are visible to the second central processor 111, and the second host port 1211 is visible to the second central processor 111. The feature that the interconnection port is hidden from the central processor is shown, for example, in the query result obtained when the operating system queries PCIe information. For example, when the instruction operating system queries the information of all PCIe ports of the PCIe bus and the information of all devices connected to the PCIe bus, the query result includes the PCIe information of the downstream port, the PCIe information of the host port, and the information of the PCIe device connected to the downstream port, but does not include the PCIe information of the interconnection port. Among them, the PCIe information includes, for example, the bus number, device number, and function number of the corresponding port.

[0228] Since the interconnection port rejects the scanning instruction during enumeration, the interconnection port is hidden from the CPU, which can achieve the effect of blocking the communication between the interconnection port and the host port. The blocked communication includes the communication from the host port to the interconnection port and the communication from the interconnection port to the host port. For example, see the attached Fig.10 For the PCIe topology tree under CPU1, since the interconnection port blocks the scanning instruction from CPU core a in CPU1, the communication between the interconnection port and the host port is blocked. For the PCIe topology tree under CPU2, since the interconnection port b blocks the scanning instruction from CPU core b, the communication between the interconnection port b and the host port b is blocked. It should be noted that blocking communication means blocking communication logically, and does not limit whether the host port and the interconnection port are physically connected or not.

[0229] By adopting the above configuration method for the interconnection port of each PCIe switch, the problem of access conflict when the PCIe switches under multiple CPUs are interconnected can be solved. The principle of this technical effect is introduced below.

[0230] During the PCIe enumeration process, since the interconnection port rejects the CPU's scan instruction, the CPU will not continue to scan the PCIe switches of other PCIe domains connected to the interconnection port from the interconnection port, and the CPU will not allocate bus identifiers for the interconnection port and the PCIe switches of other PCIe domains, thereby ensuring that the bus identifier of each PCIe device in each PCIe domain is unique. From the perspective of PCIe topology, the root node corresponding to each PCIe device in each PCIe domain is unique. Therefore, when the PCIe switches in different PCIe topology trees are interconnected, it can also ensure that multiple PCIe topology trees under multiple CPUs are independent of each other, and there will be no access conflict between different PCIe topology trees. Among them, the independence of the PCIe topology tree refers to the unique path between an endpoint and the RC, that is, starting from a PCIe device, tracing up one level at a time, a unique CPU will be found.

[0231] For example, see Appendix Figure 4 Since the first interconnection port 1202 is hidden from the first central processor 110, the first central processor 110 will not scan the second PCIe switch 121 and the second PCIe device 131 in the second PCIe domain 102 through the first interconnection port 1202 during PCIe enumeration, thereby avoiding the situation where the RC of the first PCIe domain 101 (the first central processor 110) scans the second PCIe switch 121 and the second PCIe device 131 in the second PCIe domain 102 as leaf nodes and assigns a bus identifier. Since the second interconnection port 1212 is hidden from the second central processor 111, the second central processor 111 will not scan the first PCIe switch 120 and the first PCIe device 130 in the first PCIe domain 101 through the second interconnection port 1212 during PCIe enumeration, thereby avoiding the situation where the RC of the second PCIe domain 102 (the second central processor 111) scans the first PCIe device 130 in the first PCIe domain 101 as a leaf node. Therefore, the RCs of the first PCIe domain 101 and the second PCIe domain 102 will not conflict, each PCIe device in the first PCIe domain 101 has a unique root node (the first central processor 110), and each PCIe device in the second PCIe domain 102 has a unique root node (the second central processor 111). That is, when the first PCIe switch 120 and the second PCIe switch 121 are interconnected, the PCIe topology tree under the first central processor 110 and the PCIe topology tree under the second central processor 111 are also independent of each other, and there will be no access conflict between the two PCIe topology trees.

[0232] The above introduces the configuration related to the interconnection port. The following introduces the routing information and the configuration related to the routing information.

[0233] Routing information is used to instruct the PCIe switch to perform addressing when forwarding access requests. Routing information is also called routing rules.

[0234] For example, the routing information includes the address information of the PCIe device. The address information is used to address the corresponding PCIe device. The form of the address information of the PCIe device includes various situations. For example, the address information of the first PCIe device 130 includes the address window (address window) or bus identity (bus ID, also called bus ID) of the first PCIe device 130. For example, see the attached Figure 5 and attached Figure 6 , attached Figure 6 The system 300 and the attached Figure 5 The system 200 shown has a similar structure, but the difference is that the system 200 uses an address window to represent the address of each PCIe device, while the system 300 shows the address of each PCIe device when a bus identifier is used to represent the address.

[0235] The address window includes at least one address. The address in the address window is, for example, a logical address. Different PCIe devices correspond to different address windows. The address window is used to forward access requests (i.e., P2P access requests) between different PCIe devices. Specifically, the address in the address window will be carried in the access request as the source address or destination address of the access request. The PCIe switch can determine which PCIe device the access request comes from based on the source address in the access request; the PCIe switch can determine which PCIe device the access request is to be sent to based on the destination address in the access request, thereby forwarding the access request. For example, in the attached Figure 5 In the scenario shown, the address window of NPU0 is address window 0 (address window W0), the address window of NPU1 is address window 1 (address window W1), the address window of NPU2 is address window 2 (address window W2), and the address window of NPU3 is address window 3 (address window W3). The address window of NPU4 is address window 4 (address window W4), the address window of NPU5 is address window 5 (address window W5), the address window of NPU6 is address window 6 (address window W6), and the address window of NPU7 is address window 7 (address window W7).

[0236] The bus identifier is used to identify the PCIe bus to which the PCIe device is connected. In some embodiments, the bus identifier is globally unique. Global uniqueness means that there is a one-to-one correspondence between the PCIe device and the bus identifier, and different PCIe devices have different corresponding bus identifiers. Figure 6 In the scenario shown, the bus identifier corresponding to NPU0 is bus 10, indicating that the PCIe bus connected to NPU0 is bus 10. Similarly, the bus identifier corresponding to NPU1 is bus 11, the bus identifier corresponding to NPU2 is bus 12, and the bus identifier corresponding to NPU3 is bus 13. The bus identifier corresponding to NPU4 is bus 80, the bus identifier corresponding to NPU5 is bus 81, the bus identifier corresponding to NPU6 is bus 82, and the bus identifier corresponding to NPU7 is bus 83.

[0237] Of course, the address window or bus identifier is an example of address information, and the address information is not limited to these two types of information. For example, the address information may also be a media access control (MAC) address of a PCIe device, an Internet protocol version (IP) address, or other data that can identify the identity of a PCIe device.

[0238] This embodiment does not limit which hardware in the system stores various routing information. For example, the manner of storing routing information includes the following case A and case B.

[0239] Case A: All or part of the routing information is stored in the processing unit of the PCIe switch.

[0240] The processing unit of the PCIe switch is, for example, a switch chip. The processing unit is, for example, built into the PCIe switch. The processing unit is used to control and manage various components in the PCIe switch (such as downstream ports, interconnection ports, and host ports). Figure 4 In the illustrated scenario, the first PCIe switch includes a first processing unit. The first processing unit refers to a processing unit of the first PCIe switch. For example, the first processing unit is a switch chip of the first PCIe switch. The second PCIe switch includes a second processing unit. The second processing unit refers to a processing unit of the second PCIe switch. For example, the second processing unit is a switch chip of the second PCIe switch.

[0241] Case B: Various routing information is stored in the corresponding ports of the PCIe switch.

[0242] For example, a port of a PCIe switch has a chip, and various routing information is stored in the chip of the port, so that the routing information is directly configured on the port.

[0243] When the address of a PCIe device is represented in different forms, the content of the routing information is different. For example, when the address of a PCIe device is represented in the form of an address window, the routing information is called address routing information (address routerule), and the routing information includes the address window of the PCIe device. When the address of a PCIe device is represented in the form of a bus identifier, the routing information is called ID routing information (ID route rule), and the routing information includes the bus identifier corresponding to the PCIe device.

[0244] The following uses (a) to (c) to introduce the routing information of various PCIe port configurations in the PCIe switch.

[0245] (a) Inter-domain routing information configured for interconnected ports

[0246] The cross-domain routing information is used to route the access request of the local domain to the external domain. Specifically, the cross-domain routing information indicates the correspondence between the address information of the PCIe device in the external domain and the port identifier of the interconnection port of the PCIe switch in the external domain. Therefore, the cross-domain routing information can indicate such a path: the processor or PCIe device in the local domain → the interconnection port of the PCIe switch in the local domain → the interconnection port of the PCIe switch in the external domain → the PCIe device in the external domain. Then, when the processor or PCIe device initiates access to the external domain, since the access request of the local domain can be forwarded to the external domain along such a path, the function of cross-domain access is realized.

[0247] The implementation of cross-domain routing information includes multiple situations. In some embodiments, the cross-domain routing information includes the correspondence between the domain identifier of the external domain and the port identifier of the external domain interconnection port, and the correspondence between the address information of the external domain PCIe device and the domain identifier of the external domain. In other embodiments, the cross-domain routing information includes the correspondence between the address information of the external domain PCIe device and the port identifier of the interconnection port of the external domain PCIe switch.

[0248] For example, when the address information is an address window, the cross-domain routing information includes the correspondence between the domain identifier and the port identifier of the interconnected port and the correspondence between the address window of the PCIe device and the identifier of the downstream port.

[0249] For another example, when the address information is a bus identifier, the cross-domain routing information includes the correspondence between the bus identifier corresponding to the PCIe device and the port identifier of the interconnected port, and the correspondence between the bus identifier corresponding to the PCIe device and the identifier of the downstream port.

[0250] Below, with the attached Figure 4 Taking the scenario of the two domains shown as an example, the cross-domain routing information is illustrated. In order to distinguish the cross-domain routing information describing different domains, the cross-domain routing information in the first PCIe domain is referred to as the first cross-domain routing information, and the cross-domain routing information in the second PCIe domain is referred to as the second cross-domain routing information. Among them, the first cross-domain routing information is configured on the first PCIe switch, for example. The second cross-domain routing information is configured on the second PCIe switch, for example. For example, the first cross-domain routing information is configured in the first interconnection port of the first PCIe switch. The second cross-domain routing information is configured in the second interconnection port of the second PCIe switch.

[0251] In some embodiments, the cross-domain routing information is stored in a processing unit of a PCIe switch. In other embodiments, the cross-domain routing information is stored in an interconnect port (such as a chip of the interconnect port) of a PCIe switch.

[0252] (a-1) First cross-domain routing information

[0253] The first cross-domain routing information indicates the correspondence between the address information of the second PCIe device 131 and the port identifier of the second interconnected port. For example, the first cross-domain routing information includes: the correspondence between the domain identifier of the second PCIe domain and the port identifier of the second interconnected port, and the correspondence between the address information of the second PCIe device 131 and the domain identifier of the second PCIe domain. For another example, the first cross-domain routing information includes the correspondence between the address information of the second PCIe device 131 and the port identifier of the second interconnected port.

[0254] In some embodiments, the first cross-domain routing information is stored in the first processing unit. In other embodiments, the first cross-domain routing information is stored in the first interconnect port of the first PCIe switch (eg, a chip of the first interconnect port).

[0255] In some embodiments, the first cross-domain routing information includes a first entry and a third entry.

[0256] The first entry includes a correspondence between the domain identifier of the second PCIe domain 102 and the port identifier of the second interconnect port 1212. Figure 5In the scenario shown, egress domain 2→fabric port b is configured in fabric port a of PCIe switch 1. egress domain 2→fabric port b is an example of the first entry. domain 2 is an example of a domain identifier. fabric port b is an example of a port identifier of an interconnect port. egress domain 2→fabric port b is an egress direction configuration on fabric port a, which means forwarding access requests from PCIe domain 2 to fabric port b.

[0257] By configuring the first entry on the first interconnect port 1202, when the first interconnect port 1202 forwards an access request to the outside domain (the second PCIe domain 102), the second interconnect port 1212 can be correctly addressed according to the first entry; for example, in the attached Figure 5 In the scenario shown, when interconnect port a receives an access request with domain ID "domain 2", it can correctly address interconnect port b, thereby sending the access request to interconnect port b. It can be seen that this configuration facilitates cross-domain transmission of access requests between the first PCIe domain 101 and the second PCIe domain 102.

[0258] The third entry includes a correspondence between the bus identifier corresponding to the second PCIe device 131 and the port identifier of the second interconnect port 1212. Figure 6 In the scenario shown, egress 80-83→fabric port b is configured in fabric port a of PCIe switch 1. Among them, egress 80-83→fabric port b is an example of the third entry. 80-83 is an example of the bus identifiers corresponding to the four PCIe devices. Fabric port b is an example of the port identifier of the interconnect port. Egress 80-83→fabric port b is an egress direction configuration on fabric port a, which means that the access request with the bus identifier of the destination bus belonging to (80, 83) is forwarded to fabric port b.

[0259] By configuring the third entry on the first interconnect port 1202, when the first interconnect port 1202 forwards an access request to the outside domain (the second PCIe domain 102), the second interconnect port 1212 can be correctly addressed according to the bus identifier corresponding to the destination PCIe device (the second PCIe device 131) outside the domain and the third entry; for example, in the attached Figure 5 In the scenario shown, when interconnect port a receives an access request with bus ID "80", it can correctly address interconnect port b and thus send the access request to interconnect port b.

[0260] (a-2) Second cross-domain routing information

[0261] The second cross-domain routing information indicates the correspondence between the address information of the first PCIe device 130 and the port identifier of the first interconnected port. For example, the second cross-domain routing information includes: the correspondence between the domain identifier of the first PCIe domain and the port identifier of the first interconnected port, and the correspondence between the address information of the first PCIe device 130 and the domain identifier of the first PCIe domain. For another example, the second cross-domain routing information includes the correspondence between the address information of the first PCIe device 130 and the port identifier of the first interconnected port.

[0262] In some embodiments, the second cross-domain routing information is stored in the second processing unit. In other embodiments, the second cross-domain routing information is stored in the second interconnection port of the second PCIe switch (such as a chip of the second interconnection port).

[0263] The specific content of the second cross-domain routing information includes multiple implementations. For example, the second cross-domain routing information includes a second entry and a fourth entry.

[0264] The second entry includes a correspondence between the domain identifier of the first PCIe domain 101 and the port identifier of the first interconnect port 1202. Figure 5 In the scenario shown, egress domain 1→fabric port a is configured in fabric port b of PCIe switch 2. egress domain 1→fabric port a is an example of the second entry. domain 1 is an example of a domain identifier. fabric port a is an example of a port identifier of an interconnect port. egress domain 1→fabric port a is an egress direction configuration on fabric port b, which means that access requests sent to PCIe domain 1 are forwarded to fabric port a.

[0265] By configuring the second entry on the second interconnect port 1212, when the second interconnect port 1212 forwards an access request to the outside domain (the first PCIe domain 101), it can address the first interconnect port 1202 according to the second entry. Figure 5 In the scenario shown, when interconnect port b receives an access request with domain ID "domain 1", it can correctly address interconnect port a, thereby sending the access request to interconnect port a. It can be seen that this configuration facilitates cross-domain transmission of access requests between the first PCIe domain 101 and the second PCIe domain 102.

[0266] The fourth entry includes a correspondence between a bus identifier corresponding to the first PCIe device 130 and a port identifier of the first interconnect port 1202. Figure 6 In the scenario shown, fabric port b of PCIe switch 2 is configured with egress 10-13→fabric port a. egress 10-13→fabric port a is an example of the fourth item. 10-13 is an example of the bus identifiers corresponding to the four PCIe devices. Fabric port a is an example of the port identifier of the fabric port. egress 10-13→fabric port a is an egress direction configuration on fabric port b, which means that the access request with the bus identifier of the destination bus belonging to (10, 13) is forwarded to fabric port a.

[0267] By configuring the fourth entry on the second interconnect port 1212, when the second interconnect port 1212 forwards the access request to the outside domain (the first PCIe domain 101), it can address the first interconnect port 1202 according to the bus identifier corresponding to the destination PCIe device (the first PCIe device 130) outside the domain and the fourth entry. Figure 5 In the scenario shown, when interconnect port b receives an access request with domain ID "10", it can correctly address interconnect port a, thereby sending the access request to interconnect port a. It can be seen that this configuration facilitates cross-domain transmission of access requests between the first PCIe domain 101 and the second PCIe domain 102.

[0268] (b) The local routing information of the interconnection port configuration.

[0269] The routing information of this domain is used to route access requests from external domains to this domain. Specifically, the routing information of this domain includes the correspondence between the address information of the PCIe device in this domain and the port identifier of the downstream port of the PCIe switch in this domain. Therefore, the routing information of this domain can indicate such a path: the processor or PCIe device in the external domain → the interconnection port of the PCIe switch in this domain → the downstream port of the PCIe switch in this domain → the PCIe device in this domain. Then, when the processor or PCIe device in the external domain initiates access to this domain, since the access request from the external domain can be forwarded to this domain along such a path, the function of cross-domain access is realized.

[0270] In some embodiments, the local domain routing information is stored in a processing unit of a PCIe switch. In other embodiments, the local domain routing information is stored in an interconnection port (such as a chip of the interconnection port) of a PCIe switch.

[0271] Below, with the attached Figure 4 Taking the scenario of the two domains shown as an example, the routing information of the local domain is illustrated. In order to distinguish the local domain routing information describing different domains, the local domain routing information in the first PCIe domain is referred to as the first local domain routing information, and the local domain routing information in the second PCIe domain is referred to as the second local domain routing information. Among them, the first local domain routing information is configured on the first PCIe switch, for example. The second local domain routing information is configured on the second PCIe switch, for example. For example, the first local domain routing information is configured in the first interconnection port of the first PCIe switch. The second local domain routing information is configured in the second interconnection port of the second PCIe switch.

[0272] (b-1) First domain routing information

[0273] The first local domain routing information includes the correspondence between the address information of the first PCIe device 130 and the port identifier of the first downstream port 1203. For example, the first local domain routing information includes the fifth entry and the seventh entry.

[0274] In some embodiments, the first local domain routing information is stored in the first processing unit. In other embodiments, the first local domain routing information is stored in the first interconnection port 1202 of the first PCIe switch (eg, a chip of the first interconnection port).

[0275] The fifth entry includes a correspondence between the address window of the first PCIe device 130 and the port identifier of the first downstream port 1203 .

[0276] Ingress window w0~w3→DP0~3 is an example of the fifth entry. Window w0 is an example of the address window. DP0~3 is an example of the identification of four downstream ports. Ingress window w0~w3→DP0~3 is an ingress direction configuration on fabric port a. The meaning of this configuration is to forward access requests with destination addresses belonging to window w0~w3 to the corresponding downstream ports in DP0~3.

[0277] By configuring the fifth entry on the first interconnect port 1202, when the first interconnect port 1202 receives an access request from an external domain (the second PCIe domain 102) to a destination PCIe device (the first PCIe device 130) in the domain, the first interconnect port 1202 can correctly address the downstream port (the first downstream port 1203) connected to the destination PCIe device according to the fifth entry; for example, in the attached Figure 5 In the scenario shown, when interconnect port a receives an access request with a destination address belonging to "window w0", it can correctly address DP0, thereby sending the access request to DP0 of PCIe switch 1 instead of erroneously sending it to DP1 of PCIe switch 1. It can be seen that this configuration facilitates cross-domain transmission of access requests between the first PCIe domain 101 and the second PCIe domain 102.

[0278] The seventh entry includes a correspondence between a bus identifier corresponding to the first PCIe device 130 and a port identifier of the first downstream port 1203 .

[0279] For example, in the attached Figure 6 In the scenario shown, ingress 10-13→DP0-3 is configured in fabric port a of PCIe switch 1. Ingress 10-13→DP0-3 is an example of the seventh item. 10-13 is an example of the bus identifiers corresponding to four PCIe devices. DP0-3 is an example of the identifiers of four downstream ports. Ingress 10-13→DP0-3 is an ingress configuration on fabric port a. The meaning of this configuration is to forward the access request whose bus identifier of the destination bus belongs to (10, 13) to the corresponding downstream port in DP0-3.

[0280] By configuring the seventh entry on the first interconnect port 1202, when the first interconnect port 1202 receives an access request from an external domain (the second PCIe domain 102) to a destination PCIe device (the first PCIe device 130) in the domain, the first interconnect port 1202 can correctly address the downstream port (the first downstream port 1203) connected to the destination PCIe device according to the seventh entry; for example, in the attached Figure 5 In the scenario shown, when interconnect port a receives an access request with a destination address of "10", it can correctly address DP0, thereby sending the access request to DP0 of PCIe switch 1 instead of erroneously sending it to DP1 of PCIe switch 1. It can be seen that this configuration facilitates cross-domain transmission of access requests between the first PCIe domain 101 and the second PCIe domain 102.

[0281] (b-2) Second domain routing information

[0282] The second local domain routing information includes the correspondence between the address information of the second PCIe device 131 and the port identifier of the second downstream port 1213. In some embodiments, the second local domain routing information is stored in the second processing unit. In other embodiments, the second local domain routing information is stored in the second interconnection port of the second PCIe switch (such as the chip of the second interconnection port).

[0283] For example, the second domain routing information includes the sixth entry and the eighth entry.

[0284] The sixth entry includes a correspondence between the address window of the second PCIe device 131 and the port identifier of the second downstream port 1213 .

[0285] Ingress window w4~w7→DP0~3 is an example of the sixth entry. Window w4~w7 is an example of four address windows. DP0~3 is an example of the identification of four downstream ports. Ingress windoww4~w7→DP0~3 is an ingress configuration on fabric port b (fabric port b). The meaning of this configuration is to forward access requests with destination addresses belonging to window w4~w7 to the corresponding downstream ports in DP0~3.

[0286] By configuring the sixth entry on the second interconnect port 1212, when the second interconnect port 1212 receives an access request from outside the domain (the first PCIe domain 101) to the destination PCIe device (the second PCIe device 131) in the domain, the second interconnect port 1212 can correctly address the downstream port (the second downstream port 1213) connected to the destination PCIe device according to the sixth entry.

[0287] The eighth entry includes a correspondence between a bus identifier corresponding to the second PCIe device 131 and a port identifier of the second downstream port 1213 .

[0288] Ingress 80~83→DP0~3 is an example of the eighth entry. 80~83 is an example of the bus identifiers corresponding to the four PCIe devices. DP0~3 is an example of the identifiers of the four downstream ports. Ingress 80~83→DP0~3 is an ingress configuration on fabric port b (fabric port b). The meaning of this configuration is to forward the access request with the bus identifier of the destination bus belonging to (80, 83) to the corresponding downstream port in DP0~3.

[0289] By configuring the eighth entry on the second interconnect port 1212, when the second interconnect port 1212 receives an access request from the outside domain (the first PCIe domain 101) to the destination PCIe device (the second PCIe device 131) of the domain, the second interconnect port 1212 can correctly address the downstream port (the second downstream port 1213) connected to the destination PCIe device according to the eighth entry.

[0290] Combine the above cross-domain routing information and the local domain routing information, for example, in the attached Figure 5 In the scenario shown in the figure, the interconnection port a (fabric port a) of PCIe switch 1 is configured with egress domain 2 → fabric port b, ingress window w0 ~ w3 → DP0 ~ 3. The interconnection port b (fabric port b) of PCIe switch 2 is configured with egress domain 1 → fabric port a, ingress window w4 ~ w7 → DP0 ~ 3. For another example, in the attached Figure 6 In the scenario shown, egress 80-83 → fabric port b, ingress 10-13 → DP0-3 are configured in fabric port a of PCIe switch 1. egress 10-13 → fabric port a, ingress 80-83 → DP0-3 are configured in fabric port b of PCIe switch 2.

[0291] (c) Downstream port routing information of the downstream port configuration.

[0292] The downstream port routing information is used to route access requests whose destination domain is the external domain to the corresponding interconnection port. For example, the downstream port routing information indicates the correspondence between the address information of the PCIe device in the external domain and the port identifier of the interconnection port in this domain. Therefore, the downstream port routing information can indicate such a path: the downstream port of the PCIe switch in this domain → the interconnection port of the PCIe switch in this domain → the PCIe switch in the external domain → the PCIe device in the external domain. Then, when the processor or PCIe device initiates access to the external domain, since the access request of the local domain can be forwarded from the appropriate interconnection port of the local domain along such a path, it helps the access request reach the correct external domain and realizes the function of cross-domain access.

[0293] In some embodiments, the downstream port routing information is stored in a processing unit of the PCIe switch. In other embodiments, the downstream port routing information is stored in a downstream port (eg, a chip of a downstream port) of the PCIe switch.

[0294] The implementation of the downstream port routing information includes multiple situations. In some embodiments, the downstream port routing information includes the correspondence between the domain identifier of the external domain and the port identifier of the interconnected port of the local domain, and the correspondence between the address information of the external domain PCIe device and the domain identifier of the external domain. In other embodiments, the downstream port routing information includes the correspondence between the address information of the PCIe device of the external domain and the port identifier of the interconnected port of the local domain.

[0295] The downstream port routing information is, for example, in the form of an address mapping table, also called a DP port address mapping table.

[0296] Below, with the attached Figure 4 Taking the scenario of the two domains shown as an example, the downstream port routing information is illustrated. In order to distinguish the downstream port routing information describing different domains, the downstream port routing information in the first PCIe domain 101 is referred to as the first downstream port routing information, and the downstream port routing information in the second PCIe domain is referred to as the second downstream port routing information. Among them, the first downstream port routing information is configured on the first PCIe switch 120, for example. The second downstream port routing information is configured on the second PCIe switch 121, for example. For example, the first downstream port routing information is configured in the first downstream port 1203 of the first PCIe switch 120. The second downstream port routing information is configured in the second downstream port 1213 of the second PCIe switch 121.

[0297] (c-1) First downstream port routing information

[0298] The first downstream port routing information indicates the correspondence between the address information of the second PCIe device 131 and the port identifier of the first interconnect port 1202. In some embodiments, the first downstream port routing information is stored in the first processing unit. In other embodiments, the first downstream port routing information is stored in the first downstream port of the first PCIe switch (such as a chip of the first downstream port).

[0299] The first downstream port routing information includes the relationship between the domain identifier of the second PCIe domain 102 and the port identifier of the first interconnect port 1202 (for example, the ninth item below), and the correspondence between the address information of the second PCIe device 131 and the domain identifier of the second PCIe domain 102 (for example, the eleventh item or the thirteenth item below).

[0300] The ninth entry includes a correspondence between the domain identifier of the second PCIe domain 102 and the port identifier of the first interconnection port 1202. egress domain 2→fabric port a is an example of the ninth entry. domain 2 is an example of a domain identifier. fabric port a is an example of a port identifier of an interconnection port. egress domain 2→fabric port a is a configuration of an egress direction on a downstream port, and the meaning of this configuration is to forward the access request of PCIe domain 2 to interconnection port a (fabric port a).

[0301] By configuring the ninth entry on the first downstream port 1203, when the first downstream port 1203 receives an access request whose destination domain is outside the domain (the second PCIe domain 102), the first downstream port 1203 can correctly address the first interconnection port 1202 according to the ninth entry; for example, in the attached Figure 5 In the scenario shown, when the downstream port DP0 receives an access request with domain ID "domain 2" from NPU0, the downstream port DP0 correctly addresses the interconnection port a, and the downstream port DP0 sends the access request to the interconnection port a, and the interconnection port a sends the access request to the interconnection port b, thereby forwarding the access request of the local domain to the external domain across domains. It can be seen that this configuration enables the downstream port to correctly address the corresponding interconnection port, which helps the access request to be transmitted across domains between the first PCIe domain 101 and the second PCIe domain 102.

[0302] The eleventh entry includes the correspondence between the address window of the second PCIe device 131 and the domain identifier of the second PCIe domain 102. Figure 5As shown, the eleventh entry is configured on the downstream ports (DP0 to DP3) of the PCIe switch 1, and for example, the corresponding relationship between address window w4 and egress domain 2, the corresponding relationship between address window w5 and egress domain 2, the corresponding relationship between address window w6 and egress domain 2, and the corresponding relationship between address window w7 and egress domain 2. By configuring the eleventh entry on the first downstream port, the PCIe switch 1 can find that the PCIe domain to which the access request is to be sent is the PCIe domain 2 according to the address window of the PCIe device outside the domain.

[0303] The thirteenth entry includes the correspondence between the bus identifier corresponding to the second PCIe device 131 and the port identifier of the first interconnection port 1202. egress 80~83→fabric port a is an example of the thirteenth entry. 80~83 is an example of the interval to which the bus identifier of the device in the PCIe domain 2 belongs. Fabric port a is an example of the port identifier of the interconnection port. Egress 80~83→fabric port a is an entry direction configuration on the downstream port, and the meaning of this configuration is to forward the access request whose bus identifier of the destination bus belongs to (80, 83) to the interconnection port a (fabric port a).

[0304] (c-2) Second downstream port routing information

[0305] The second downstream port routing information indicates the correspondence between the address information of the first PCIe device 130 and the port identifier of the second interconnected port. In some embodiments, the second downstream port routing information is stored in the second processing unit. In other embodiments, the second downstream port routing information is stored in the second downstream port of the second PCIe switch 121 (such as the chip of the second downstream port).

[0306] The second downstream port routing information includes the relationship between the domain identifier of the first PCIe domain and the port identifier of the second interconnected port (for example, the tenth item below), and the correspondence between the address information of the first PCIe device 130 and the domain identifier of the second PCIe domain (for example, the twelfth item or the fourteenth item below).

[0307] The tenth entry includes the correspondence between the domain identifier of the first PCIe domain 101 and the port identifier of the second interconnect port 1212. egress domain 1→fabric port b is an example of the tenth entry. domain 2 is an example of the domain identifier. fabric port b is an example of the port identifier of the interconnect port. egress domain 1→fabric port b is an egress direction configuration on the downstream port, and the meaning of this configuration is to forward the access request sent to PCIe domain 2 to interconnect port b (fabric port b). By configuring the tenth entry on the second downstream port 1213, when the second downstream port 1213 receives an access request whose destination domain is outside the domain (the first PCIe domain 101), the second downstream port 1213 can correctly address the second interconnect port 1212 according to the tenth entry.

[0308] The twelfth entry includes the correspondence between the address window of the first PCIe device 130 and the domain identifier of the first PCIe domain 101. Figure 5 As shown, the twelfth entry is configured on the downstream ports (DP0 to DP3) of the PCIe switch 2, and includes, for example, the correspondence between address window w1 and egress domain 1, the correspondence between address window w2 and egress domain 1, the correspondence between address window w3 and egress domain 1, and the correspondence between address window w4 and egress domain 1. Through the twelfth entry on the second downstream port, the PCIe switch 2 can find that the PCIe domain to which the access request is to be sent is PCIe domain 1 according to the address window of the PCIe device outside the domain.

[0309] egress 10~13→fabric port b is an example of the fourteenth item. 10~13 is an example of the interval to which the bus identifier of the device in PCIe domain 1 belongs. Fabric port b is an example of the port identifier of the interconnection port. egress10~13→fabric port b is an ingress configuration on the downstream port. The meaning of this configuration is to forward the access request with the bus identifier of the destination bus belonging to (10, 13) to the interconnection port b (fabric port b).

[0310] The above describes some routing information related to cross-domain access and specific items in the routing information. This embodiment does not limit the downstream port or interconnection port to only configure the above-listed routing information. Optionally, the downstream port or interconnection port also configures other routing information to further improve communication reliability. Figure 4 The scenes shown and the attached Figure 5 The scenario shown provides examples of some routing information that may be configured on downstream ports or interconnected ports.

[0311] For example, see Appendix Figure 4 , in the attached Figure 4 In the illustrated system 100, the first downstream port 1203 is configured with the thirteenth entry, the fifteenth entry, and the seventeenth entry, and the second downstream port 1213 is configured with the fourteenth entry, the sixteenth entry, and the eighteenth entry. Figure 5 , in the attached Figure 5 In the system with two CPUs and eight NPUs shown in the figure, the address entries of the downstream port of PCIe switch 1 include egress domain 2→fabric port a, ingress window w0→NPU0, ingress window w1→NPU1, ingress window w2→NPU2, ingress window w3→NPU3. The address entries of the downstream port of PCIe switch 2 include egress domain 1→fabric port b, ingress window w4→NPU4, ingress window w5→NPU5, ingress window w6→NPU6, ingress window w7→NPU7. For another example, see the attached Figure 6 , in the attached Figure 6 In the scenario shown, the ID routing information of the downstream ports of PCIe switch 1 includes egress 80-83→fabric port a, ingress 10→NPU0, ingress 11→NPU1, ingress 12→NPU2, and ingress 13→NPU3. The ID routing information of the downstream ports of PCIe switch 2 includes egress 10-13→fabricport b, ingress 80→NPU4, ingress 81→NPU5, ingress 82→NPU6, and ingress 83→NPU7.

[0312] The fourteenth entry includes a correspondence between a bus identifier corresponding to the first PCIe device 130 and a port identifier of the second interconnect port 1212 .

[0313] The fifteenth entry includes a correspondence between the address window of the first PCIe device 130 and the identifier of the first PCIe device 130 .

[0314] Ingress window w1→NPU1 is an example of the fifteenth item. Window w1 is an example of the address window. NPU1 is an example of the identification of the PCIe device. Ingress window w1→NPU1 is an ingress direction configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w1 (address windoww1) to NPU1.

[0315] Ingress window w2→NPU2 is an example of the fifteenth item. Window w2 is an example of the address window. NPU2 is an example of the identification of the PCIe device. Ingress window w2→NPU2 is an ingress direction configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w2 (address windoww2) to NPU2.

[0316] Ingress window w3→NPU3 is an example of the fifteenth item. Window w3 is an example of the address window. NPU3 is an example of the identification of the PCIe device. Ingress window w3→NPU3 is an ingress configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w3 (address window w3) to NPU3.

[0317] The sixteenth entry includes a correspondence between the address window of the second PCIe device 131 and the identifier of the second PCIe device 131 .

[0318] Ingress window w4→NPU4 is an example of the sixteenth entry. Window w4 is an example of the address window. NPU4 is an example of the identification of the PCIe device. Ingress window w4→NPU4 is an ingress configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w4 (address window w4) to NPU4.

[0319] Ingress window w5→NPU5 is an example of the sixteenth entry. Window w5 is an example of the address window. NPU5 is an example of the identification of the PCIe device. Ingress window w5→NPU5 is an ingress configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w5 (address windoww5) to NPU5.

[0320] Ingress window w6→NPU6 is an example of the sixteenth entry. Window w6 is an example of the address window. NPU6 is an example of the identification of the PCIe device. Ingress window w6→NPU6 is an ingress direction configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w6 (address window w6) to NPU6.

[0321] Ingress window w7→NPU7 is an example of the sixteenth entry. Window w7 is an example of the address window. NPU7 is an example of the identification of the PCIe device. Ingress window w7→NPU7 is an ingress configuration on the downstream port. The meaning of this configuration is to forward the access request whose destination address belongs to address window w7 (address window w7) to NPU7.

[0322] The seventeenth entry includes a correspondence between the bus identifier corresponding to the first PCIe device 130 and the identifier of the first PCIe device 130 .

[0323] Ingress 10→NPU0 is an example of the seventeenth item. 10 is an example of a bus identifier. NPU0 is an example of a PCIe device identifier. Ingress 10→NPU0 is an ingress configuration, which means that the access request with the destination bus being bus 10 is forwarded to NPU0.

[0324] ingress 11→NPU1 is an example of the seventeenth item. 11 is an example of a bus identifier. NPU1 is an example of a PCIe device identifier. ingress 11→NPU1 is an ingress configuration, which means that the access request with the destination bus being bus 11 is forwarded to NPU1.

[0325] ingress 12→NPU2 is an example of the seventeenth item. 12 is an example of a bus identifier. NPU2 is an example of a PCIe device identifier. ingress 12→NPU2 is an ingress configuration, which means that the access request with the destination bus being bus 12 is forwarded to NPU2.

[0326] ingress 13→NPU3 is an example of the seventeenth item. 13 is an example of a bus identifier. NPU3 is an example of a PCIe device identifier. ingress 13→NPU3 is an ingress configuration, which means that the access request with the destination bus being bus 13 is forwarded to NPU3.

[0327] The eighteenth entry includes a correspondence between the bus identifier corresponding to the second PCIe device 131 and the identifier of the second PCIe device 131 .

[0328] ingress 80→NPU4 is an example of the eighteenth item. 80 is an example of a bus identifier. NPU4 is an example of a PCIe device identifier. ingress 80→NPU4 is an ingress configuration, which means that the access request with the destination bus being bus 80 is forwarded to NPU4.

[0329] ingress 81→NPU5 is an example of the eighteenth item. 81 is an example of a bus identifier. NPU5 is an example of a PCIe device identifier. ingress 81→NPU5 is an ingress configuration, which means that the access request with the destination bus being bus 81 is forwarded to NPU5.

[0330] ingress 82→NPU6 is an example of the eighteenth item. 82 is an example of a bus identifier. NPU6 is an example of a PCIe device identifier. ingress 82→NPU6 is an ingress configuration, which means that the access request with the destination bus being bus 82 is forwarded to NPU6.

[0331] ingress 83→NPU7 is an example of the eighteenth item. 83 is an example of a bus identifier. NPU7 is an example of a PCIe device identifier. ingress 83→NPU7 is an ingress configuration, which means that the access request with the destination bus being bus 83 is forwarded to NPU7.

[0332] The above describes various routing information that may be configured on a PCIe port. The following describes a method flow for configuring the routing information described above on a PCIe port of a PCIe switch.

[0333] This embodiment does not limit the timing of configuring routing information to the PCIe port. In some embodiments, the step of configuring routing information to the PCIe port is performed after the system is powered on and before the task is started. For example, the step of configuring routing information is performed during the system initialization phase. For example, in the scenario of AI training, before starting the AI ​​training task, the step of configuring routing information is performed so that in the process of executing AI training, the pre-configured routing information is applied to forward cross-domain P2P access requests of different AI training cards.

[0334] This embodiment does not limit the execution subject of configuring routing information to the PCIe port. In some embodiments, the BIOS is responsible for configuring routing information on the PCIe port of the PCIe switch. Figure 4 In the scenario shown, the BIOS configures at least one of the first cross-domain routing information or the first local domain routing information to the first interconnect port 1202; the BIOS configures the first downstream port routing information to the first downstream port 1203; and the BIOS configures the second downstream port routing information to the second downstream port 1213.

[0335] In some embodiments, the action of BIOS configuring routing information (such as at least one of cross-domain routing information or local domain routing information) to the interconnection port is implemented by interacting with the management port of the PCIe switch. Figure 7 , the BIOS running on the CPU generates a configuration instruction and sends the configuration instruction to the management port, and the configuration instruction includes routing information. The PCIe switch receives the configuration instruction through the management port. The PCIe switch obtains the routing information from the configuration instruction and configures the routing information to the interconnection port. In some embodiments, the action of configuring the routing information to the interconnection port is specifically implemented by accessing the register of the interconnection port. For example, the management port sends the configuration instruction to the firmware in the PCIe switch, the firmware in the PCIe switch obtains the routing information from the configuration instruction, writes the routing information to the register of the interconnection port, and the register of the interconnection port saves the routing information. When receiving an access request, the interconnection port reads the routing information from the register and forwards the access request according to the routing information.

[0336] It is worth noting that, when the routing information is configured by the BIOS, this embodiment does not limit which CPU configures the routing information by running the BIOS. Figure 4 Taking the system architecture shown in the figure as an example, the CPU configuring routing information to the PCIe port includes but is not limited to the following cases A to D.

[0337] Case A: BIOS includes multiple threads, and different threads of BIOS run on different central processing units. Optionally, in this case, different central processing units configure routing information on different PCIe ports through different threads of BIOS. For example, BIOS includes a first thread and a second thread, the first thread runs on the first central processing unit 110, and the second thread runs on the second central processing unit 111. Optionally, in this case, the first central processing unit 110 configures corresponding routing information to the first interconnection port 1202 and the first downstream port 1203 through the first thread, and the second central processing unit 111 configures corresponding routing information to the second interconnection port 1212 and the second downstream port 1213 through the second thread.

[0338] Case B: BIOS runs on the first central processor 110. Optionally, in this case, the first central processor 110 configures corresponding routing information to the first interconnection port 1202, the first downstream port 1203, the second interconnection port 1212 and the second downstream port 1213 respectively through BIOS.

[0339] Case C: BIOS runs on the second central processor 111. Optionally, in this case, the second central processor 111 configures corresponding routing information to the first interconnection port 1202, the first downstream port 1203, the second interconnection port 1212 and the second downstream port 1213 respectively through BIOS.

[0340] Case D: BIOS runs on other CPUs other than the first CPU 110 and the second CPU 111. Optionally, in this case, other CPUs configure corresponding routing information to the first interconnection port 1202, the first downstream port 1203, the second interconnection port 1212, and the second downstream port 1213 through BIOS.

[0341] It is worth noting that the case where the first central processor 110 configures the first interconnection port 1202 is an example. When the first PCIe switch 120 has multiple interconnection ports, the first central processor 110 configures corresponding routing information to each interconnection port of the first PCIe switch 120. In this way, each interconnection port can forward to the correct interconnection port of other PCIe domains in the egress direction. Each interconnection port can forward to the correct downstream port in the ingress direction. The steps of configuring other interconnection ports by the first central processor 110 are similar to the steps of configuring the first interconnection port 1202.

[0342] It is worth noting that the case where the first central processor 110 configures the first downstream port 1203 is an example. When the first PCIe switch 120 has multiple downstream ports, the first central processor 110 configures corresponding routing information to each downstream port of the first PCIe switch 120. In this way, each downstream port can forward to the correct interconnection port in the egress direction. Each downstream port can forward to the correct PCIe device in the ingress direction. The steps for the first central processor 110 to configure other downstream ports are similar to the steps for configuring the first downstream port 1203.

[0343] In some embodiments, the routing information is executed by an address mapper run by a central processor. For example, the configuration steps include: the address mapper obtains the domain identifier of each PCIe domain in the multi-processor system and the BAR space address of the PCIe device in each PCIe domain, and saves the corresponding relationship between the BAR space address of the PCIe device and the domain identifier to the address window mapping table. Then, the address mapper identifies the P2P access relationship between the PCIe devices, and configures the mapping relationship between the address windows of each pair of PCIe devices that need to access each other to the downstream port and the interconnection port.

[0344] The BAR space address is, for example, a device space access address assigned by the BIOS according to the PCIe protocol after completing the PCIe device scan. The BAR space address is, for example, a memory-mapped I / O (MMIO) address. The address window mapping table is, for example, cached in the CPU memory.

[0345] The P2P access relationship is determined, for example, according to business needs. For example, in the scenario of AI training, if the system has 20 AI training cards, training task A requires the use of 8 AI training cards, and training task B requires the use of 12 AI training cards, then a P2P access relationship is established for the 8 AI training cards, and a P2P access relationship is established for the 12 AI training cards. In addition, the 8 AI training cards corresponding to training task A and the 12 AI training cards corresponding to training task B may not establish a P2P access relationship.

[0346] The action of configuring the port is implemented by, for example, writing a register. For example, each PCIe port has a corresponding configuration register, and the mapping relationship between address windows is written into the configuration register so that the configuration register stores the mapping relationship between address windows, thereby implementing the configuration port.

[0347] The above describes the routing information of each PCIe port configuration in the PCIe switch. Figure 4, describes the process of a PCIe switch using configured routing information to open a P2P access channel in a cross-domain P2P access process.

[0348] Please refer to the attached Figure 4 , taking the scenario of the first PCIe device 130 and the second PCIe device 131 performing P2P access as an example, an example is given to illustrate how the first PCIe switch 120 implements cross-domain P2P access. Cross-domain P2P access involves a processing flow in the egress direction and a processing flow in the ingress direction. The following example illustrates how the first PCIe switch 120 executes the processing flow in the egress direction through scenario one; and an example illustrates how the first PCIe switch 120 executes the processing flow in the ingress direction through scenario two. Among them, the first PCIe device 130 and the second PCIe device 131 belong to different PCIe domains, and the first PCIe device 130 belongs to the first PCIe domain 101. The first PCIe device 130 is connected to the first downstream port 1203 of the first PCIe switch 120. The second PCIe device 131 belongs to the second PCIe domain 102. The second PCIe device 131 is connected to the second downstream port 1213 of the second PCIe switch 121.

[0349] Scenario 1: The first PCIe device 130 accesses the second PCIe device 131 .

[0350] Specifically, the first PCIe device 130 generates a first access request and sends the first access request to the first PCIe switch 120. The source PCIe device of the first access request is the first PCIe device 130, and the destination PCIe device of the first access request is the second PCIe device 131 in the second PCIe domain 102. After the first PCIe switch 120 receives the first access request, the task of the first PCIe switch 120 is to send the first access request from the first PCIe domain 101 to the second PCIe domain 102, that is, routing forwarding in the egress direction.

[0351] In some embodiments, the PCIe switch performs an action of adding the domain identifier of other PCIe domains in the access request. For example, after the first PCIe switch 120 receives the first access request through the first downstream port 1203, the first PCIe switch 120 determines the domain identifier of the second PCIe domain 102 according to the first access request and the routing information configured by the first downstream port 1203. The first PCIe switch 120 adds the domain identifier of the second PCIe domain 102 in the first access request; the first PCIe switch 120 sends the first access request including the domain identifier to the second interconnection port 1212 through the first interconnection port 1202.

[0352] Specifically, the routing information configured by the first downstream port 1203 includes the correspondence between the address window and the domain identifier. The first PCIe switch 120 determines the destination address of the first access request, compares the destination address with each address window in the routing information configured by the first downstream port 1203, and determines the domain identifier corresponding to the address window to which the destination address belongs. If the determined domain identifier is the domain identifier of the second PCIe domain 102, indicating that the destination address of the access request is mapped to other domains, the first PCIe switch 120 adds the domain identifier of the second PCIe domain 102 to the header of the destination address, so that the first access request includes the domain identifier. Afterwards, the first downstream port 1203 sends the first access request including the domain identifier to the first interconnect port 1202.

[0353] Through the above method, when the PCIe switch receives an access request to a PCIe device in another PCIe domain, it will add the domain identifier of the other PCIe domain in the access request, so that the interconnection port can quickly match whether the access request is to access the PCIe device in the current domain according to the added domain identifier, thereby simplifying the table lookup action.

[0354] In some embodiments, during cross-domain P2P access, the downstream port will address the corresponding interconnected port according to the configured routing information. The following uses implementation A to illustrate how the downstream port addresses the interconnected port when addressing according to the address window, and uses implementation B to illustrate how the downstream port addresses the interconnected port when addressing according to the bus identifier.

[0355] Implementation A: After the first PCIe switch 120 receives the first access request through the first downstream port 1203, the first PCIe switch 120 determines the first interconnection port 1202 according to the first access request and the ninth entry configured by the first downstream port 1203. Then, the first downstream port 1203 sends the first access request to the determined first interconnection port 1202.

[0356] The ninth entry corresponds to the downstream port routing information. The ninth entry includes the correspondence between the domain identifier of the second PCIe domain 102 and the port identifier of the first interconnected port 1202. The first PCIe switch 120 queries the ninth entry according to the domain identifier to obtain the port identifier of the first interconnected port 1202, and determines the first interconnected port 1202 according to the port identifier of the first interconnected port 1202.

[0357] For example, in the attached Figure 5In the scenario shown, the downstream port is configured with egress domain 2→fabric port a. The first PCIe switch 120 queries this configuration according to the domain identifier domain 2, and obtains the port identifier as fabric port a, and then determines that the interconnection port a identified by fabric port a is the interconnection port connected to domain 2. Afterwards, the downstream port sends a first access request to the interconnection port a. In this example, egress domain 2→fabric port a is the ninth entry configured by the first downstream port 1203.

[0358] Since the first downstream port 1203 is configured with the ninth entry, the first downstream port 1203 can use the ninth entry to send the access request sent to the second PCIe domain 102 to the first interconnection port 1202, which helps to send the access request from the first interconnection port 1202 to the second PCIe domain 102, realizing PCIe cross-domain address mapping.

[0359] In some embodiments, the domain identifiers of other PCIe domains are found based on entries configured in the downstream ports. For example, the first downstream port 1203 is configured with an eleventh entry, and the first PCIe switch 120 determines the domain identifier of the second PCIe domain 102 based on the first access request and the eleventh entry.

[0360] Among them, the eleventh entry corresponds to the downstream port routing information. The eleventh entry includes the correspondence between the address window of the second PCIe device 131 and the domain identifier of the second PCIe domain 102. The eleventh entry is an example of the correspondence between the out-of-domain address window and the domain identifier. The first PCIe switch 120 queries the eleventh entry according to the destination address in the first access request. In the process of querying the eleventh destination, the first PCIe switch 120 matches the address window of each PCIe device according to the destination address. When the address window of the second PCIe device 131 is matched, since the address window of the second PCIe device 131 corresponds to the domain identifier of the second PCIe domain 102, it is determined that the domain identifier to be added is the domain identifier of the second PCIe domain 102.

[0361] Implementation B: After the first PCIe switch 120 receives the first access request through the first downstream port 1203, the first PCIe switch 120 determines the first interconnection port 1202 according to the first access request and the thirteenth item configured by the first downstream port 1203. Then, the first downstream port 1203 sends the first access request to the determined first interconnection port 1202.

[0362] The thirteenth entry corresponds to the downstream port routing information. The thirteenth entry includes the correspondence between the bus identifier corresponding to the second PCIe device 131 and the port identifier of the first interconnected port 1202. The first PCIe switch 120 queries the thirteenth entry according to the bus identifier corresponding to the second PCIe device 131, obtains the port identifier of the first interconnected port 1202, and determines the first interconnected port 1202 according to the port identifier of the first interconnected port 1202. The first access request includes the bus identifier corresponding to the second PCIe device 131.

[0363] For example, see Appendix Figure 6 , the downstream port is configured with egress 80~83→fabric port a. The second PCIe device 131 is NPU4, and the bus identifier corresponding to the second PCIe device 131 is 80. The first PCIe switch 120 queries this configuration according to 80, and obtains the port identifier as fabric port a, and then determines the interconnection port a identified by fabric port a. Afterwards, the downstream port sends a first access request to the interconnection port a. In this example, egress 80~83→fabric port a is the thirteenth item configured in the first downstream port 1203.

[0364] In some embodiments, when the interconnection port receives an access request, the interconnection port forwards the access request to the interconnection port of the PCIe switch of the other PCIe domain according to the configured entry. For example, after the interconnection port of the first PCIe switch 120 receives the first access request, the first PCIe switch 120 determines the second interconnection port 1212 according to the first access request and the first entry configured for the first interconnection port 1202.

[0365] In some embodiments, the first entry corresponds to cross-domain routing information. The first entry includes a correspondence between a domain identifier of the second PCIe domain 102 and a port identifier of the second interconnected port 1212. The first PCIe switch 120 queries the first entry according to the domain identifier of the second PCIe domain 102, obtains the port identifier of the second interconnected port 1212, and determines the second interconnected port 1212 according to the port identifier of the second interconnected port 1212.

[0366] For example, see Appendix Figure 5 , interconnection port a is configured with egress domain 2→fabric port b. The first PCIe switch 120 queries this configuration according to the domain identifier domain 2 and obtains the port identifier as fabric port b. Afterwards, interconnection port a sends a first access request to interconnection port b identified by fabric port b.

[0367] By writing the correspondence between the domain identifier and the port identifier of the interconnection port in the configuration of the interconnection port, when the interconnection port forwards the cross-domain P2P access request, the access request can be transparently transmitted to the interconnection port of the other domain according to the configuration.

[0368] Scenario 2: The second PCIe device 131 accesses the first PCIe device 130 .

[0369] Specifically, the second PCIe device 131 generates a second access request and sends the second access request to the second PCIe switch 121. The source PCIe device of the second access request is the second PCIe device 131 in the second PCIe domain 102, and the destination PCIe device of the second access request is the first PCIe device 130. The second PCIe switch 121 executes the routing forwarding process in the egress direction and forwards the second access request to the first PCIe switch 120. After the first PCIe switch 120 receives the second access request, the task of the first PCIe switch 120 is to send the second access request to the downstream PCIe device, that is, routing forwarding in the ingress direction. Among them, the routing forwarding process in the egress direction executed by the second PCIe switch 121 is the same as the routing forwarding process in the egress direction executed by the first PCIe switch 120 in the scenario below.

[0370] In some embodiments, the routing forwarding in the ingress direction is implemented based on the interaction between the interconnection port and the downstream port in the PCIe switch. Specifically, after the second access request is sent out from the second interconnection port 1212 of the second PCIe switch 121, it will reach the first interconnection port 1202 of the first PCIe switch 120. After the second PCIe switch 121 receives the second access request through the first interconnection port 1202, the second PCIe switch 121 will determine the first downstream port 1203. The first interconnection port 1202 sends the second access request to the first downstream port 1203. After the first downstream port 1203 receives the second access request, the first downstream port 1203 sends the second access request to the first PCIe device 130.

[0371] In some embodiments, during cross-domain P2P access, the interconnection port will address the corresponding downstream port according to the configured routing information. The following uses implementation A to illustrate how the interconnection port addresses the downstream port when addressing according to the address window, and uses implementation B to illustrate how the downstream port addresses the interconnection port when addressing according to the bus identifier.

[0372] Implementation A: After the first PCIe switch 120 receives the second access request through the first interconnection port 1202, the first PCIe switch 120 determines the first downstream port 1203 according to the second access request and the fifth entry. Then, the first interconnection port 1202 sends the first access request to the determined first downstream port 1203.

[0373] Among them, the fifth entry corresponds to the routing information of this domain. The fifth entry includes the correspondence between the address window of the first PCIe device 130 and the port identifier of the first downstream port 1203. The destination address in the second access request belongs to the address window of the first PCIe device 130. The first PCIe switch 120 matches the pre-configured address window according to the destination address in the second access request, determines that the destination address in the second access request belongs to the address window of the first PCIe device 130, queries the fifth entry according to the address window of the first PCIe device 130, obtains the port identifier of the first downstream port 1203, and determines the first downstream port 1203 according to the port identifier of the first downstream port 1203.

[0374] It can be seen from the above process that since the interconnection port pre-configures the correspondence between the address window and the port identifier of the downstream port, when the interconnection port receives an access request in the inlet direction (that is, an access request to the PCIe device in the domain sent by the interconnection port of the other domain), the interconnection port can forward the access request to the corresponding downstream port according to the address carried in the access request, and then route it to the corresponding PCIe device.

[0375] Implementation B: After the first PCIe switch 120 receives the second access request through the first interconnection port 1202, the first PCIe switch 120 determines the first downstream port 1203 according to the second access request and the seventh entry configured by the first interconnection port 1202. Then, the first interconnection port 1202 sends the first access request to the determined first downstream port 1203.

[0376] The seventh entry corresponds to the routing information of the local domain. The seventh entry includes the correspondence between the bus identifier corresponding to the first PCIe device 130 and the port identifier of the first downstream port 1203. The first PCIe switch 120 determines the first downstream port 1203 according to the second access request and the seventh entry. The second access request includes the bus identifier corresponding to the first PCIe device 130.

[0377] In some embodiments, the PCIe device in the domain to which the ingress access request is forwarded is determined according to the configuration of the downstream port.

[0378] When addressing according to the address window, the downstream port is, for example, configured with a correspondence between the address window of the PCIe device and the identifier of the PCIe device. For example, the first downstream port 1203 is configured with the fifteenth entry. When the first PCIe switch 120 receives the second access request, it determines the first PCIe device 130 according to the second access request and the fifteenth entry configured for the first downstream port 1203. Among them, the fifteenth entry is the address routing information. The fifteenth entry includes the correspondence between the address window of the first PCIe device 130 and the identifier of the first PCIe device 130. Specifically, the first PCIe switch 120 obtains the destination address from the second access request, queries the configuration of the first downstream port 1203 according to the destination address, determines that the destination address hits the address window in the fifteenth entry, obtains the identifier of the first PCIe device 130 from the fifteenth entry, and thereby determines the first PCIe device 130. For example, see the attached Figure 5 , the configuration of the downstream port includes the correspondence between the address window and the AI ​​training card. For example, the configuration of the downstream port includes ingress window w0→NPU0. If the destination address of the received access request belongs to window w0, after PCIe switch 1 queries the configuration of the downstream port, it will forward the access request to NPU0 through downstream port DP0. In this way, when the downstream port processes the access request in the inlet direction, it can automatically address the specified PCIe device according to the pre-configured address entry and the destination address of the access request, thereby forwarding the received access request to the PCIe device.

[0379] When addressing according to the bus identifier, the downstream port is, for example, configured with a correspondence between the bus identifier corresponding to the PCIe device and the identifier of the PCIe device. For example, the first downstream port 1203 is configured with the seventeenth entry. When the first PCIe switch 120 receives the second access request, it determines the first PCIe device 130 according to the second access request and the seventeenth entry configured for the first downstream port 1203. Among them, the seventeenth entry is ID routing information. The seventeenth entry includes the correspondence between the bus identifier corresponding to the first PCIe device 130 and the identifier of the first PCIe device 130. Specifically, the first PCIe switch 120 obtains the bus identifier of the destination bus from the second access request, queries the configuration of the first downstream port 1203 according to the bus identifier of the destination bus, determines that the bus identifier of the destination bus hits the bus identifier in the seventeenth entry, obtains the identifier of the first PCIe device 130 from the seventeenth entry, and thus determines the first PCIe device 130. For example, see the attached Figure 6, the configuration of the downstream port includes the correspondence between the bus identifier and the AI ​​training card. For example, the configuration of the downstream port includes ingress 10→NPU0. If the bus identifier of the destination bus of the received access request is 10, after the PCIe switch 1 queries the configuration of the downstream port, it will forward the access request to NPU0 connected to bus 10 through the downstream port DP0. In this way, when the downstream port processes the access request in the inlet direction, it can automatically address the specified PCIe device according to the pre-configured ID entry and the bus identifier of the destination bus of the access request, thereby forwarding the received access request to the PCIe device.

[0380] By configuring routing information to the PCIe port, the PCIe switch forwards the access request using the configured routing information, thereby constructing a P2P channel and an H2D channel in the multi-processor system. The following uses (I) to (II) to introduce the P2P channel and the H2D channel in the system architecture of this embodiment.

[0381] (I) P2P Channel

[0382] The multi-processor system includes multiple P2P channels. A P2P channel refers to a channel between different PCIe devices. The P2P channel is used to transmit access requests between different PCIe devices. The starting point of the P2P channel is a PCIe device (such as an AI training card), and the end point of the P2P channel is another PCIe device (such as another AI training card). The P2P channel is established based on a PCIe switch. The P2P channel includes at least one of an intra-domain P2P channel or a cross-domain P2P channel.

[0383] (I-1) Intra-domain P2P channel

[0384] Intra-domain P2P channels refer to P2P channels located in the same PCIe domain. Intra-domain P2P channels are used to transmit access requests between different PCIe devices in the same PCIe domain. The starting point and end point of the intra-domain P2P channel are two PCIe devices in the same PCIe domain. The intra-domain P2P channel is established based on the PCIe switch in the domain. Figure 4 In the dual-CPU system shown, the intra-domain P2P channel includes a first intra-domain P2P channel and a second intra-domain P2P channel.

[0385] The first intra-domain P2P channel refers to the intra-domain P2P channel between different PCIe devices in the first PCIe domain 101. The first intra-domain P2P channel can be represented as a PCIe device in the first PCIe domain → first PCIe switch 120 → another PCIe device in the first PCIe domain. The starting point and end point of the first intra-domain P2P channel are two PCIe devices in the first PCIe domain 101, respectively. The first intra-domain P2P channel is established through the first PCIe switch 120. When a PCIe device in the first PCIe domain sends an access request, the first PCIe switch 120 forwards the access request to another PCIe device 130, so that the access request is transmitted to another PCIe device in the first PCIe domain through the first intra-domain P2P channel. For example, see the attached Figure 5 , NPU0→PCIe switch 1→NPU1 This channel is an example of the first H2D channel.

[0386] The second intra-domain P2P channel refers to the intra-domain P2P channel between different PCIe devices in the second PCIe domain 102. The second intra-domain P2P channel can be represented as a PCIe device in the second PCIe domain → second PCIe switch 121 → another PCIe device 130 in the second PCIe domain. The starting point and the end point of the second intra-domain P2P channel are two PCIe devices in the second PCIe domain 102, respectively. The second intra-domain P2P channel is established through the second PCIe switch 121. When a PCIe device in the second PCIe domain sends an access request, the second PCIe switch 121 forwards the access request to another PCIe device in the second PCIe domain, so that the access request is transmitted to another PCIe device in the second PCIe domain through the second intra-domain P2P channel. For example, see the attached Figure 5 , NPU0→PCIe switch 1→NPU1 This channel is an example of the second H2D channel.

[0387] (I-2) Cross-domain P2P channel

[0388] A cross-domain P2P channel is a P2P channel between PCIe devices in different PCIe domains. A cross-domain P2P channel is used to transmit access requests from a PCIe device in one PCIe domain to a PCIe device in another PCIe domain. The starting point of a cross-domain P2P channel is a PCIe device in one PCIe domain, and the starting point of a cross-domain P2P channel is a PCIe device in another PCIe domain. For example, see the attached Figure 5, NPU0→PCIe switch 1→PCIe switch 2→NPU4 is an example of a cross-domain P2P channel. The cross-domain P2P channel is established based on PCIe switches in multiple PCIe domains. The PCIe switch forwards the access request of the PCIe device in the domain to the PCIe switches in other PCIe domains, so that the access request can be transmitted across domains.

[0389] See attached Figure 4 , taking the cross-domain transmission from the first PCIe domain 101 to the second PCIe domain 102 as an example, the cross-domain P2P channel includes the first PCIe device 130→the first PCIe switch 120→the second PCIe switch 121→the second PCIe device 131. When the first PCIe device 130 initiates an access request to the PCIe device 131, the first PCIe switch 120 forwards the access request to the second PCIe switch 121, so that the access request is transmitted from the first PCIe domain 101 to the second PCIe domain 102. The second PCIe switch 121 forwards the access request to the second PCIe device 131, and the second PCIe device 131 receives the access request from the second PCIe switch 121.

[0390] See attached Figure 4 , taking the cross-domain transmission from the second PCIe domain 102 to the first PCIe domain 101 as an example, the cross-domain P2P channel includes the second PCIe device 131→the second PCIe switch 121→the first PCIe switch 120→the first PCIe device 130. When the second PCIe device 131 initiates an access request to the first PCIe device 130, the second PCIe switch 121 forwards the access request to the first PCIe switch 120, so that the access request is transmitted from the second PCIe domain 102 to the first PCIe domain 101. The first PCIe switch 120 forwards the access request to the first PCIe device 130, and the first PCIe device 130 receives the access request from the first PCIe switch 120.

[0391] The multiple P2P channels provided by the above system architecture can significantly improve the P2P communication performance between different PCIe devices. The technical principle for achieving this technical effect is introduced below.

[0392] On the one hand, in system 100 or system 200, since the cross-domain P2P channel is established based on PCIe switches in different PCIe domains, the cross-domain P2P access request is forwarded by multiple PCIe switches. Therefore, the cross-domain P2P access request can fully utilize the high-speed transmission capability of the PCIe switch when forwarding, reducing the transmission delay of the cross-domain P2P access request. In addition, since the transmission of the cross-domain P2P access request does not rely on the forwarding of the CPU, the situation where the heavy CPU load affects the performance of the cross-domain P2P communication is avoided, thereby improving the reliability and stability of the cross-domain P2P communication. For example, compared with the attached Figure 2 and attached Figure 5 , in the attached Figure 2 In the manner shown, since the NPU under PCIe switch 1 relies on CPU1 and CPU2 for forwarding when accessing the NPU under PCIe switch 2, once CPU1 or CPU2 is heavily loaded, the P2P communication performance between NPUs will be affected. Figure 5 In the manner shown, since the NPU under PCIe switch 1 accesses the NPU under PCIe switch 2, PCIe switch 1 and PCIe switch 2 are responsible for forwarding, and do not rely on CPU1 and CPU2. Therefore, when NPUs under different PCIe switches access each other, they will not be affected by the high load of CPU1 or CPU2, thereby ensuring the reliability of communication performance when the NPUs under PCIe switch 1 and PCIe switch 2 access each other.

[0393] On the other hand, in the system 100 or the system 200, since the task of forwarding cross-domain P2P access requests is offloaded from the CPU to multiple PCIe switches, the load on the CPU is reduced.

[0394] In particular, when this method is applied to a single-machine multi-card AI training scenario, since this method ensures that the P2P communication performance of different AI training cards in the server is consistent, the P2P communication performance of the AI ​​training cards is balanced, which helps different AI training cards to quickly synchronize parameters through the P2P channel.

[0395] (II) H2D channel

[0396] The multi-processor system includes multiple H2D channels. The H2D channel refers to a channel between a central processor and a PCIe device. The H2D channel is used to transmit the access request of the central processor to the PCIe device. The starting point of the H2D channel is, for example, the central processor. The end point of the H2D channel is, for example, the PCIe device. The H2D channel is established based on a PCIe switch. Figure 4 In the dual-CPU system shown, the H2D channel includes a first H2D channel and a second H2D channel.

[0397] The first H2D channel refers to the H2D channel between the first central processor 110 and the first PCIe device 130. The first H2D channel can be represented as the first central processor 110→the first PCIe switch 120→the first PCIe device 130. The starting point of the first H2D channel is the first central processor 110. The end point of the first H2D channel is the first PCIe device 130. The first H2D channel is established based on the first PCIe switch 120. When the first central processor 110 sends an access request, the first PCIe switch 120 forwards the access request to the first PCIe device 130, so that the access request is transmitted to the first PCIe device 130 through the first H2D channel. For example, see the attached Figure 5 , attached Figure 5 The channel of CPU1→PCIe switch 1→NPU0 is an example of the first H2D channel.

[0398] The second H2D channel refers to the H2D channel between the second central processor 111 and the second PCIe device 131. The second H2D channel can be represented as the second central processor 111→the second PCIe switch 121→the second PCIe device 131. The starting point of the second H2D channel is the second central processor 111. The end point of the second H2D channel is the second PCIe device 131. The second H2D channel is established based on the second PCIe switch 121. When the second central processor 111 sends an access request, the second PCIe switch 121 forwards the access request to the second PCIe device 131, so that the access request is transmitted to the second PCIe device 131 through the second H2D channel. For example, see the attached Figure 5 , the channel of CPU2→PCIe switch 2→NPU4 is an example of the second H2D channel.

[0399] The multiple H2D channels provided by the above system architecture can significantly improve the H2D communication performance of multiple CPUs. The technical principle for achieving this technical effect is introduced below.

[0400] On the one hand, in system 100 or system 200, since the H2D channels of different CPUs are established through different PCIe switches, the access requests of different CPUs will be forwarded through different PCIe switches. Therefore, different CPUs can use the bandwidth resources and processing resources provided by different PCIe switches when performing H2D access, thereby avoiding the situation where the bandwidth resources and processing resources required for H2D access by different CPUs occupy each other, ensuring the balance of bandwidth resources and processing resources required for H2D access by each CPU, so that the H2D communication performance of multiple CPUs remains balanced. For example, compared with the attached Figure 1 and attached Figure 5 , in the attached Figure 5In the method shown, since CPU1's access request to NPU passes through PCIe switch 1, CPU2's access request to NPU also passes through PCIe switch 1, CPU2 and CPU1 both occupy the bandwidth and processing resources of PCIe switch 1 when performing H2D access, resulting in insufficient resources for a single CPU to perform H2D access. Figure 1 In the manner shown, since CPU1's access request to NPU passes through PCIe switch 1, CPU2's access request to NPU passes through PCIe switch 2, CPU2 and CPU1 use bandwidth resources and processing resources provided by different PCIe switches when performing H2D access and will not affect each other. Therefore, the H2D communication performance of CPU2 accessing the PCIe device will be consistent with the H2D communication performance of CPU1 accessing the PCIe device.

[0401] On the other hand, in system 100 or system 200, since different H2D channels are independent of each other, the access request of each CPU does not need to rely on the forwarding of CPUs in other PCIe domains, thereby avoiding the situation where a single point failure of a CPU causes other CPUs to be unable to perform H2D access, thereby improving the reliability of H2D communication; and since the task of forwarding H2D access requests is basically offloaded to the PCIe switch, the CPU is relieved of the load of forwarding H2D access requests for CPUs in other PCIe domains.

[0402] For example, compared with Figure 1 and attached Figure 5 , in the attached Figure 1 In the method shown, since CPU1 is responsible for forwarding H2D access requests for CPU2, the CPU1 load will be heavier. In addition, once CPU1 fails or the link between CPU1 and CPU2 fails, CPU2 may not be able to access H2D, resulting in insufficient reliability. Figure 5 In the illustrated manner, the PCIe switch 2 is responsible for forwarding the H2D access request of CPU2 without passing through CPU1, thereby ensuring load balancing between CPU1 and CPU2, and avoiding the situation where CPU2 cannot access H2D due to a single point failure of CPU1, thereby improving the reliability of H2D communication of CPU2.

[0403] In summary, in the system and method provided by this embodiment, since the routing information corresponding to each PCIe domain is configured for various ports of the PCIe switch, PCIe cross-domain address mapping is implemented, and the interconnection of the PCIe switch and the configured routing information can be used to open up cross-domain P2P channels between multiple independent PCIe topology trees, thereby ensuring that PCIe cross-domain P2P access has the same performance as intra-domain P2P access.

[0404] It is worth noting that the various routing information listed above is an example of a PCIe switch interconnecting with another PCIe switch through an interconnection port. When a PCIe switch is interconnected with PCIe switches in multiple PCIe domains through multiple interconnection ports, the system can optionally configure similar routing information for each interconnection port and each downstream port in the same manner as the various routing information listed above. Figure 5 In the scenario shown, optionally, the PCIe switch 1 includes not only the interconnection port a, but also the interconnection port c. Figure 5 The interconnection port d of the PCIe switch 3 of the unshown PCIe domain 3 is connected. In this case, similarly to the cross-domain routing information (such as the first entry), the interconnection port c is configured with egress domain 3→fabric port d, for example; similarly to the downstream port routing information (the ninth entry), the downstream ports DP0 to DP3 are also configured with egress domain 3→fabric port ac, for example.

[0405] By analogy, those skilled in the art can understand that the configuration scheme of the routing information described above can be used as a general solution for realizing cross-domain P2P access under a multi-CPU, multi-PCIe device architecture. For example, when the PCIe switch 2 under CPU2, the PCIe switch 3 under CPU3... the PCIe switch i under CPUi... the PCIe switch n under CPUn are interconnected to form n PCIe domains to collaboratively process tasks, by configuring the corresponding routing information to the interconnection port and downstream port of each PCIe switch in PCIe switch 2, PCIe switch 3, PCIe switch i, and PCIe switch n, it is possible to open a cross-domain P2P channel between any two or more PCIe domains in PCIe domain 1, PCIe domain 2... PCIe domain i... PCIe domain n. In this way, when it is necessary to coordinate the computing power of a large number of CPUs and a large number of PCIe devices to perform AI training or other computing tasks, this method helps to flexibly transmit access requests across a large number of PCIe domains, thereby facilitating high-speed data synchronization between a large number of CPUs and a large number of PCIe devices, and meeting the requirements of load symmetry between different CPUs and different PCIe devices.

[0406] The following is an example of how to apply a multi-processor system to an AI training scenario.

[0407] In some embodiments, the multi-processor system is used in a scenario where a single machine has multiple cards for AI training. Multiple cards means that the multi-processor system has multiple AI training cards. In an exemplary scenario, according to the business plan, a first number of CPUs and a second number of AI training cards are connected through a PCIe switch, and a symmetrical load training system is formed through the configuration method provided in this embodiment, and training is performed through the symmetrical load training system. For example, see the attached Figure 5 , connect 2 CPUs and 8 NPUs through 2 PCIe switches to form a dual-CPU, 8-node scenario.

[0408] Optionally, a multi-processor system is applied in combination with the technology of thread binding CPU core. Specifically, a corresponding thread is assigned to each PCIe device, and a binding relationship between the thread and the CPU core is established to meet the affinity requirements of the non-uniform memory access architecture (NUMA). When a multi-processor system executes a task, the thread will run on the bound CPU core and will not be scheduled to other CPU cores. The loads of each thread are completely symmetrical, so that they are basically started and completed synchronously, avoiding long waits for each other, and maximizing the efficiency of the multi-processor system. Among them, when applied in the scenario of AI training, the thread is, for example, a training thread.

[0409] By applying the method provided by this embodiment to the AI ​​training scenario, the AI ​​training card can be equipped with high-speed P2P communication capabilities, and the AI ​​training cards under different PCIe switches can quickly synchronize parameters through cross-domain P2P channels, thereby accelerating the execution of AI training tasks. In addition, since the cross-domain P2P access of different PCIe devices is realized through the PCIe switch, and does not rely on the dedicated bus for P2P communication, on the one hand, the cost of adding a dedicated bus for P2P communication is eliminated, so the implementation cost is low. On the other hand, it solves the limitation problem that the PCIe device must have an interface for accessing the dedicated bus for P2P communication to access cross-domain P2P, and can be applied to various PCIe devices, so the application scenarios are wide and practical. In addition, since the method ensures that the H2D communication performance of different CPUs in the server to the AI ​​training card is equal, the H2D communication performance of multiple CPUs is balanced, which helps each CPU and the AI ​​training card to perform training tasks such as data copying and message synchronization more quickly.

[0410] In some embodiments, a computer device is also provided, the computer device comprising an attached Figure 4 The system 100 shown in FIG. Figure 5 The system 200 shown or attached Figure 6 The system 300 is shown. The computer device is, for example, a server, a mainframe or a personal computer.

[0411] In some embodiments, a device for configuring a multi-processor system is provided. Figure 4 The system 100 shown in FIG. Figure 5 The system 200 shown or attached Figure 6 The system 300 shown in FIG. 1 is a system for configuring a multi-processor system. The device for configuring a multi-processor system includes at least one unit. In some embodiments, the unit in the device for configuring a multi-processor system is implemented by software, and the unit in the device provided by the fourth aspect is a program module. In other embodiments, the unit in the device for configuring a multi-processor system is implemented by hardware or firmware. For specific details of the device for configuring a multi-processor system, please refer to the above-mentioned Appendix. Figure 4 The system 100 shown in FIG. Figure 5 The system 200 shown in FIG. Figure 6 The system 300 shown in FIG. Figure 8 The software 400 shown, Figure 5 The corresponding description of the method 500 shown will not be repeated here.

[0412] In some embodiments, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is read by a processor to configure an attached Figure 4 The system 100 shown in FIG. Figure 5 The system 200 shown or attached Figure 6 System 300 is shown.

[0413] In some embodiments, a computer program product is provided, the computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a multi-processor system reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to configure an attached Figure 4 The system 100 shown in FIG. Figure 5 The system 200 shown or attached Figure 6 System 300 is shown.

[0414] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0415] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0416] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0417] The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0418] In addition, each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or software unit.

[0419] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0420] In the present application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first PCIe switch may be referred to as a second PCIe switch, and similarly, a second PCIe switch may be referred to as a first PCIe switch. Both the first PCIe switch and the second PCIe switch may be PCIe switches, and in some cases, may be separate and different PCIe switches.

[0421] The term "at least one" in this application means one or more, and the term "multiple" in this application means two or more, for example, multiple PCIe devices means two or more PCIe devices. The terms "system" and "network" are often used interchangeably herein.

[0422] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0423] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0424] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0425] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0426] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0427] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multiprocessor system, characterized in that: The system includes a first peripheral component interconnect express (PCIe) domain and a second PCIe domain; The first PCIe domain includes a first processor, a first PCIe switch connected to the first processor, and a first PCIe device connected to the first PCIe switch; The second PCIe domain includes a second processor, a second PCIe switch connected to the second processor, and a second PCIe device connected to the second PCIe switch, and the first PCIe switch communicates with the second PCIe switch via a bus; When the first processor performs enumeration, the second PCIe device will not be discovered by the first processor; The first processor or the first PCIe device accesses the second PCIe device through the bus.

2. The system according to claim 1, characterized in that The first PCIe switch includes a first interconnection port, the first interconnection port is used to communicate with the second PCIe switch, and the first interconnection port is configured to reject a scan instruction of the first processor when the first processor is enumerated.

3. The system according to claim 2, characterized in that The first PCIe switch includes firmware configured to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

4. The system according to claim 2, characterized in that The system further includes a basic input and output system BIOS, which is used to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

5. The system according to claim 2, characterized in that The system also includes a baseboard management controller BMC; The BMC is used to configure the first interconnect port to reject the scan instruction of the first processor when the first processor is enumerated.

6. The system according to any one of claims 2 to 5, characterized in that: The second PCIe switch comprises a second interconnection port, and the second interconnection port is used to communicate with the first PCIe switch; The first interconnect port is configured with cross-domain routing information, and the first processor or the first PCIe device accesses the second PCIe device through the cross-domain routing information, and the cross-domain routing information indicates the correspondence between the address information of the second PCIe device and the port identifier of the second interconnect port.

7. The system according to claim 6, characterized in that The cross-domain routing information includes: The corresponding relationship between the domain identifier of the second PCIe domain and the port identifier of the second interconnect port, and the corresponding relationship between the address information of the second PCIe device and the domain identifier of the second PCIe domain.

8. The system according to claim 6, characterized in that The first PCIe switch includes a processing unit, and the cross-domain routing information is stored in the processing unit.

9. The system according to any one of claims 2-5, 7-8, characterized in that: The first PCIe switch comprises a first downstream port, wherein the first downstream port is connected to the first PCIe device; The first interconnect port is configured with local domain routing information, and the local domain routing information includes a correspondence between the address information of the first PCIe device and the port identifier of the first downstream port.

10. The system according to any one of claims 2-5, 7-8, characterized in that: The first PCIe switch includes a first downstream port, in which downstream port routing information is configured, and the downstream port routing information indicates a correspondence between address information of the second PCIe device and a port identifier of the first interconnect port.

11. The system according to claim 10, characterized in that The downstream port routing information includes a relationship between a domain identifier of the second PCIe domain and a port identifier of the first interconnect port, and a corresponding relationship between address information of the second PCIe device and a domain identifier of the second PCIe domain.

12. The system according to any one of claims 6, characterized in that: The address information of the first PCIe device includes an address window or a bus identifier of the first PCIe device.

13. The system according to any one of claims 2-5, 7-8, 11-12, characterized in that: The first PCIe switch further includes a first host port, and the first PCIe switch is connected to the first processor via the first host port, wherein: The first host port is not directly connected to the first interconnect port.

14. A method for configuring a multi-processor system, characterized in that: The method comprises: A first peripheral component interconnect express (PCIe) domain and a second PCIe domain are provided, wherein the first PCIe domain includes a first processor, a first PCIe switch connected to the first processor, and a first PCIe device connected to the first PCIe switch; the second PCIe domain includes a second processor, a second PCIe switch connected to the second processor, and a second PCIe device connected to the second PCIe switch; the first PCIe switch communicates with the second PCIe switch via a bus; The first PCIe domain and the second PCIe domain are configured so that when the first processor performs enumeration, the second PCIe device is not discovered by the first processor, wherein the first processor or the first PCIe device accesses the second PCIe device through the bus.

15. The method according to claim 14, characterized in that The first PCIe switch comprises a first interconnection port, and the first interconnection port is used to communicate with the second PCIe switch. Configuring the first PCIe domain and the second PCIe domain includes: The first interconnect port is configured so that the first interconnect port rejects a scan instruction of the first processor when the first processor is enumerated.

16. The method according to claim 15, characterized in that The first PCIe switch includes firmware, and the operation of configuring the first interconnect port is performed by the firmware.

17. The method according to claim 15, characterized in that The operation of configuring the first interconnect port is performed by a basic input and output system BIOS.

18. The method according to claim 15, characterized in that The operation of configuring the first interconnect port is performed by a baseboard management controller BMC.

19. The method according to any one of claims 15 to 18, characterized in that: The second PCIe switch comprises a second interconnection port, and the second interconnection port is used to communicate with the first PCIe switch; The configuring the first PCIe domain and the second PCIe domain includes: Cross-domain routing information is configured for the first interconnect port, and the first processor or the first PCIe device accesses the second PCIe device through the cross-domain routing information, wherein the cross-domain routing information is used to indicate a correspondence between address information of the second PCIe device and a port identifier of the second interconnect port.

20. The method according to claim 19, characterized in that The cross-domain routing information includes: The corresponding relationship between the domain identifier of the second PCIe domain and the port identifier of the second interconnect port, and the corresponding relationship between the address information of the second PCIe device and the domain identifier of the second PCIe domain.

21. The method according to claim 19, characterized in that The first PCIe switch includes a processing unit, and the cross-domain routing information is stored in the processing unit.

22. The method according to any one of claims 15-18, 20-21, characterized in that: The first PCIe switch comprises a first downstream port, wherein the first downstream port is connected to the first PCIe device; The configuring the first PCIe domain and the second PCIe domain includes: Local domain routing information is configured for the first interconnect port, where the local domain routing information includes a correspondence between address information of the first PCIe device and a port identifier of the first downstream port.

23. The method according to any one of claims 15-18, 20-21, characterized in that: The first PCIe switch comprises a first downstream port, The configuring the first PCIe domain and the second PCIe domain includes: Downstream port routing information is configured for the first downstream port, where the downstream port routing information indicates a correspondence between address information of the second PCIe device and a port identifier of the first interconnect port.

24. The method according to claim 23, characterized in that The downstream port routing information includes a relationship between a domain identifier of the second PCIe domain and a port identifier of the first interconnect port, and a corresponding relationship between address information of the second PCIe device and a domain identifier of the second PCIe domain.

25. The method according to any one of claims 19, characterized in that: The address information of the first PCIe device includes an address window or a bus identifier of the first PCIe device.

26. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein at least one instruction is stored in the memory. When the at least one instruction is executed on the computer device, the computer device executes the method according to any one of claims 14 to 25.

27. A computer-readable storage medium, characterized in that: At least one instruction is stored in the storage medium. When the at least one instruction is executed on a computer device, the computer device executes the method according to any one of claims 14 to 25.

Images