A PCIe switch with multiple non-transparent bridge ports and its communication method

Abstract

This invention discloses a PCIe switch with multiple non-transparent bridge ports, including non-transparent bridge ports and a main control module. The non-transparent bridge ports are used to parse PCIe data packets and set interrupt states according to feedback from the main control module. The main control module is communicatively connected to the non-transparent bridge ports and includes a doorbell register and / or temporary register for the non-transparent bridge ports to monitor the resource occupancy status of the non-transparent bridge ports, thereby effectively avoiding conflicts caused by multiple root unions accessing the same root union resource at the same time.

Description

Technical Field

[0001] This invention relates to the field of computer communication technology, and in particular to a PCIe switch with multiple non-transparent bridge ports and its communication method. Background Technology

[0002] Peripheral Component Interconnect Express (PCIe) is a local bus technology and interface standard. PCIe devices use an independent address space, namely the address space of the PCIe bus, also known as the PCIe bus domain. The PCIe protocol defines three types of devices: Root Complex (RC), switch, and Endpoint (EP). PCIe switches can expand PCIe interfaces, interconnecting different devices with PCIe interfaces for information exchange and processing. Internally, a PCIe switch contains multiple PCI-PCI bridges, often referred to as transparent bridges. Devices connected to a transparent bridge belong to the same PCIe bus domain. Because the PCIe protocol specifies that there is only one RC in the tree topology of a PCIe bus domain, all ports in a PCIe bus domain, except for the RC, are usually connected to EPs.

[0003] Within the same PCIe bus domain, EPs can transmit data using peer-to-peer (P2P) transmission without the involvement of RCs. Two RCs, however, need to be connected via a PCIe Non-Transparent Bridging (NTB). The NTB isolates the address spaces of different PCIe bus domains to connect different RCs; in this case, the NTB simultaneously acts as a PCIe device in both connected PCIe bus domains. Ports with NTB functionality typically contain resources such as doorbell registers and scratch pad registers. The doorbell registers are used for sending interrupt information between RCs, while the scratch pad registers provide command words for exchanging small amounts of information.

[0004] A PCIe switch with multiple NTB ports can connect multiple Control Registries (RCs). RCs can exchange information by accessing each other's doorbell registers and temporary registers. However, in a PCIe switch with multiple NTB ports, if multiple RCs simultaneously access the same RC's resources and perform write operations, the multiple write requests may overwrite each other, resulting in information loss or errors in subsequent information processing. Summary of the Invention

[0005] To address some or all of the problems in the prior art, the first aspect of this invention provides a PCIe switch with multiple non-transparent bridge ports, comprising:

[0006] A non-transparent bridge port is used to connect to the RC and parse PCIe packets to obtain the address information carried in the packets, and to set the interrupt status based on feedback from the main control module; and

[0007] The main control module includes a doorbell register and / or temporary register for the non-transparent bridge port, and the main control module is communicatively connected to the non-transparent bridge port. The main control module is used to monitor the resource occupancy status of the non-transparent bridge port, especially the root union connected to the non-transparent bridge port.

[0008] Furthermore, the non-transparent bridge port includes a packet monitoring module, which is used to parse and obtain the address information carried in the PCIe data packet.

[0009] Furthermore, the packet monitoring module includes addresses of resource mappings for other non-transparent bridge ports, wherein the resources include doorbell registers and / or temporary registers.

[0010] Furthermore, the non-transparent bridge port includes an interrupt handling module, which is used to set the interrupt status according to the feedback from the main control module.

[0011] A second aspect of the present invention provides a communication method for a PCIe switch as described above, wherein each root union connected to the PCIe switch can access each other, wherein a root union connected to the PCIe switch can exchange information by performing a write operation on a doorbell register or temporary register corresponding to another root union connected to the PCIe switch, the write operation including the steps of:

[0012] The root consortium sends a memory write packet and generates a write request via a non-transparent bridge port, which is then sent to the main control module; and

[0013] The main control module queries the resource usage of the target root union based on the target address in the write request:

[0014] If the resources corresponding to the target root union are occupied, the main control module sends an error status to the non-transparent bridge port and does not perform a write operation; and

[0015] If the resource corresponding to the target root union is not occupied, the data in the write request will be written to the resource corresponding to the target root union.

[0016] Furthermore, the write operation also includes the following steps:

[0017] After receiving an error status, the non-transparent bridge port sets an interrupt status and assembles an interrupt message packet, such as Message Signaled Interrupt (MSI) or MSI-X, and sends it to the root union.

[0018] Furthermore, the write operation also includes the following steps:

[0019] After receiving the interrupt message packet, the root consortium sends a memory write packet to clear the interrupt flag, and waits for a specified time before resending the memory write packet to generate a write request.

[0020] Furthermore, the generation of a write request includes:

[0021] After receiving a memory write packet, the non-transparent bridge port parses the target address in the memory write packet and sends a write request to the main control module based on the target address.

[0022] Furthermore, the non-transparent bridge port parses the memory write packets through the packet monitoring module.

[0023] Furthermore, the target address includes the mapping address of the resource corresponding to the target root union on the current non-transparent bridge port.

[0024] Furthermore, the non-transparent bridge port is set to an interrupt state through an interrupt handling module, and an interrupt message packet is formed.

[0025] This invention provides a PCIe switch with multiple non-transparent bridge ports and its communication method. First, a packet monitoring module is set up on each non-transparent bridge port to parse PCIe data packets. Then, an interrupt handling module is also set up on each non-transparent bridge port, which can set an interrupt status and form an interrupt message packet such as MSI or MSI-X to notify the control unit (RC) when a target resource is occupied. Whether the target resource is occupied is monitored by the main control module. This automatic hardware judgment and feedback of the target resource status effectively reduces the query packets sent to check the status, simplifies the software-level process, and improves efficiency and speed. Attached Figure Description

[0026] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0027] Figure 1 This diagram illustrates a structural schematic of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention; and

[0028] Figure 2 This diagram illustrates a write operation between root federations of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention. Detailed Implementation

[0029] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details. Furthermore, it should be understood that the embodiments shown in the drawings are illustrative representations and are not necessarily drawn to scale.

[0030] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0031] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0032] In this invention, the modules according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor.

[0033] To address the conflict issue arising when multiple root complexes (RCs) access the same RC's resources in a PCIe switch with multiple non-transparent bridging (NTB) ports, this invention provides a PCIe switch with multiple NTB ports and its communication method. The PCIe switch connects to multiple RCs via multiple NTB ports, where the addresses of resources on each NTB port, such as doorbell registers or temporary registers, are mapped into the address spaces (bar spaces) of other NTB ports. Simultaneously, each NTB port has a packet monitoring module to monitor PCIe standard memory read packets or memory write packets, and to parse the addresses in these packets to determine if they fall within the address spaces of resources on other NTB ports. When an address in a memory read or write packet falls within the address space range of a particular NTB port, the packet monitoring module sends a corresponding read / write request to the main control module and waits for a status flag. Furthermore, to simplify the software-level process and improve efficiency and speed, this application primarily utilizes hardware for automatic status determination and feedback. Specifically, an interrupt handling module is first set up on each NTB port. When the interrupt handling module receives an error status returned by the main control module, it can set an interrupt status and notify the corresponding root union. The main control module includes doorbell registers or temporary registers for each port, simultaneously handles read / write requests for each NTB port, monitors any conflicts that arise, and provides feedback on the status of each read / write request.

[0034] The present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0035] Figure 1 This diagram illustrates the structure of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention. Figure 1 As shown, a PCIe switch with multiple non-transparent bridge ports includes a main control module 101 and N non-transparent bridge ports 121 to 12N. As illustrated, the main control module 101 can be communicatively connected to all the non-transparent bridge ports 121 to 12N. Here, N is a natural number greater than 1.

[0036] The non-transparent bridge ports 121 to 12N are used to isolate the address spaces of different PCIe bus domains to connect different RC 001 to 00N. In an embodiment of the present invention, the non-transparent bridge ports 121 to 12N can parse the PCIe standard data packets sent by the RC 001 to 00N to obtain the address information carried within them. In addition, the non-transparent bridge ports 121 to 12N can also set corresponding states and form interrupt message packets, such as Message Signaled Interrupt (MSI) or MSI-X message packets, and notify the root union. As mentioned above, in one embodiment of the present invention, the Bar space of each NTB port contains the mapped addresses of all other NTB port resources, thereby enabling it to determine the final sending target of the PCIe standard data packets based on the parsed address information. The PCIe standard data packets include memory read packets and memory write packets.

[0037] In one embodiment of the present invention, the monitoring and parsing of the PCIe standard data packets by the NTB port is implemented through a packet monitoring module, as shown in Figures 1211 to 12N1, where each NTB port includes a packet monitoring module. The packet monitoring module is equipped with packet monitoring logic for parsing and obtaining the address information carried in the PCIe data packets. In one embodiment of the present invention, the packet monitoring module also extracts data from the PCIe data packets, and can then send the extracted data to the main control module along with the write request. In one embodiment of the present invention, the packet monitoring logic is implemented entirely at the hardware level. However, it should be understood that in other embodiments of the present invention, the packet monitoring module may also implement this function using software, firmware, or a combination of software and hardware.

[0038] In one embodiment of the present invention, the interrupt message packet is composed of an interrupt handling module, as shown in Figures 1212 to 12N2. Each NTB port includes an interrupt handling module. After receiving an error status returned by the main control module 101, the interrupt handling module sets an interrupt status and simultaneously assembles an interrupt message packet to return to the RC, reporting the write error status. In one embodiment of the present invention, the interrupt handling module is implemented entirely at the hardware level. However, it should be understood that in other embodiments of the present invention, the packet monitoring module may also implement this function using software, firmware, or a combination of software and hardware.

[0039] The main control module 101 includes the resources of the NTB port, namely the doorbell register and / or temporary storage register of each NTB port, and can monitor the resource occupancy of the NTB port, especially the RC connected to the NTB port, thereby avoiding the conflict problem of multiple RCs accessing the same resource.

[0040] The RCs connected to the PCIe switch can access each other's corresponding doorbell registers and temporary registers to exchange information. Generally, multiple RCs simultaneously accessing the same resource and performing write operations can easily cause access conflicts. Therefore, Figure 2 This diagram illustrates a write operation between root federations of a PCIe switch with multiple non-transparent bridge ports, according to an embodiment of the present invention. Figure 2 As shown, the write operation includes:

[0041] First, in step 201, a write request is generated. When an RC needs to access another RC and perform a write operation, it first sends a memory write packet to the NTB port, and then generates a write request through the NTB port and sends it to the main control module. In one embodiment of the present invention, after receiving the memory write packet, the NTB port parses the target address in the memory write packet through its packet monitoring module, and sends a write request to the main control module according to the target address. In one embodiment of the present invention, the target address refers to the mapping address of the resource corresponding to the target RC on the current NTB port. In one embodiment of the present invention, the packet monitoring module also extracts data from the PCIe data packet, and can then send the extracted data to the main control module along with the write request.

[0042] Next, in step 202, the resource usage status is queried. After receiving the write request, the main control module determines the target RC based on the target address in the write request and obtains the resource usage status corresponding to the target RC. If the resource corresponding to the target RC is occupied, the main control module returns an error status, indicating that the write has failed, and proceeds to step 231 to interrupt the operation. If the resource corresponding to the target RC is not occupied, then step 204, data writing, can proceed.

[0043] In step 231, an interrupt operation is performed. After receiving the error status returned by the main control module, the NTB port sets the interrupt status and assembles an interrupt message packet, such as a Message Signaled Interrupt (MSI) or MSI-X message packet, and returns it to the RC to report the write error status. In one embodiment of the present invention, the NTB port sets the interrupt status and assembles an interrupt message packet through an interrupt handling module.

[0044] Next, in step 232, the interrupt is cleared. After receiving the interrupt message packet, the RC sends a memory write packet to clear the interrupt flag, and waits for a specified time before resending to step 201 to send a memory write packet to generate a write request. Furthermore, if the RC does not receive an interrupt message packet after sending the memory write packet, it indicates that the resources corresponding to its target RC are idle, and therefore it can continue to send new memory write packets to continue sending new data to the target RC; and

[0045] In step 204, data is written. When the resource corresponding to the target RC is idle, the main control module writes the data in the write request to the resource corresponding to the target RC, completing the write operation. Furthermore, to better ensure communication security and prevent the RC from mistakenly interpreting a crash or message reception failure as a successful write, in one embodiment of the present invention, an MSI or MSI-X message packet is also sent to notify the RC after successful data writing.

[0046] Perform the write operation according to the steps described above until all data is sent and the information exchange between RCs is completed. During this process, the status of the target resource is mainly determined and fed back automatically by the hardware, which is more efficient than sending query packets.

[0047] Furthermore, in practical applications, multiple RCs may send write requests simultaneously. To address this issue, in one embodiment of the present invention, when the main control module receives multiple write requests simultaneously, it will prioritize the request sent by the NTB port with the highest priority according to a preset priority. In one embodiment of the present invention, the preset priority refers to sorting ports from largest to smallest. In another embodiment of the present invention, the preset priority refers to sorting ports from smallest to largest. In yet another embodiment of the present invention, the preset priority can also be customized by the user according to actual needs.

[0048] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A PCIe switch with multiple non-transparent bridge ports, characterized in that, include: A non-transparent bridge port is configured to connect to the root union and parse PCIe packets, and set an interrupt status based on feedback from the master control module. The non-transparent bridge port includes an interrupt handling module, which is configured to set an interrupt status and notify the corresponding root union when it receives an error status returned by the master control module. as well as The main control module includes a doorbell register and / or a temporary register for the non-transparent bridge port, and the main control module is communicatively connected to the non-transparent bridge port. The main control module is configured to monitor the resource occupancy status of the non-transparent bridge port, and if the resource is occupied, it returns an error status.

2. The PCIe switch as described in claim 1, characterized in that, The non-transparent bridge port includes a packet monitoring module, which is configured to parse and obtain address information from the PCIe data packets.

3. The PCIe switch as described in claim 1, characterized in that, The address space of the non-transparent bridge port includes the addresses of resource mappings of other non-transparent bridge ports, wherein the resources include doorbell registers and / or temporary registers.

4. The communication method of the PCIe switch as described in any one of claims 1 to 3, characterized in that, A root federation connected to the PCIe switch exchanges information by writing to resources corresponding to another root federation connected to the PCIe switch, wherein the write operation includes the steps of: The root consortium sends a memory write packet and generates a write request through the non-transparent bridge port, which is then sent to the main control module. The main control module queries the resource usage of the target root union based on the target address in the write request: If the resources corresponding to the target root union are occupied, the main control module sends an error status to the non-transparent bridge port and does not perform a write operation. as well as If the resource corresponding to the target root union is not occupied, the data in the write request is written to the resource corresponding to the target root union; as well as After receiving an error status, the non-transparent bridge port sets the interrupt status through the interrupt handling module and sends an interrupt message packet to the root union.

5. The communication method as described in claim 4, characterized in that, It also includes the following steps: After receiving the interrupt message packet, the root consortium sends a memory write packet to clear the interrupt flag, and waits for a specified time before resending the memory write packet to regenerate the write request.

6. The communication method as described in claim 4, characterized in that, The non-transparent bridge port parses the memory write packets through the packet monitoring module, wherein the memory write packets are PCIe standard data packets.

7. The communication method as described in claim 6, characterized in that, The target address includes the mapping address of the resource corresponding to the target root union on the current non-transparent bridge port.

Images