A PCIe switch with multiple non-transparent bridge ports and communication method thereof

Abstract

The present invention discloses a PCIe switch with multiple non-transparent bridge ports, which includes a non-transparent bridge port and a main control module, wherein the non-transparent bridge port is used to parse PCIe data packets, the main control module is communicatively connected to the non-transparent bridge port, and includes a doorbell register and / or a temporary register of the non-transparent bridge port, which is used to uniformly manage access rights between the non-transparent bridge ports, thereby effectively avoiding conflicts caused by multiple root complexes accessing the same root complex resource at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer communications, and in particular to a PCIe switch with multiple non-transparent bridge ports and a communication method thereof. Background Art

[0002] Peripheral Component Interconnect Express (PCIe) is a local bus technology and interface standard. PCIe devices use an independent address space, namely the address space of the PCIe bus, also known as the PCIe bus domain. The PCIe protocol specifies three types of devices, including the root complex (RC), switch, and end point (EP). The PCIe switch can extend the PCIe interface and interconnect different devices with PCIe interfaces for information exchange and processing. The PCIe switch contains multiple PCI-PCI bridges, which are usually called transparent bridges. The devices hung on a transparent bridge all belong to the same PCIe bus domain. Since the PCIe protocol specifies that there is only one RC in the tree topology of the same PCIe bus domain, in a PCIe bus domain, except for the RC, other ports are usually connected to the EP.

[0003] EPs in the same PCIe bus domain can use peer-to-peer (P2P) transmission to transmit data without the participation of RC. Two RCs need to be connected through a PCIe non-transparent bridge (NTB). NTB can isolate the address space of different PCIe bus domains to connect different RCs. At this time, NTB also acts as a PCIe device for the two PCIe bus domains connected to it. Ports with NTB function usually contain some resources, such as doorbell registers and scratch pad registers. Doorbell registers are used to send interrupt information between RCs, and scratch pad registers are used to provide some command words to exchange small amounts of information.

[0004] A PCIe switch with multiple NTB ports can connect multiple RCs, and RCs can exchange information by accessing each other's doorbell registers and registers. However, in a PCIe switch with multiple NTB ports, if multiple RCs access the resources of the same RC at the same time and perform write operations, multiple write requests accessed at the same time may overwrite each other, resulting in information loss or information processing errors. Summary of the invention

[0005] In view of some or all of the problems in the prior art, the first aspect of the present invention provides a PCIe switch with multiple non-transparent bridge ports, comprising:

[0006] A non-transparent bridge port, which is used to connect to the RC and parse the PCIe data packet to obtain the address information carried in the data packet; and

[0007] A main control module, which includes a doorbell register and / or a temporary register of the non-transparent bridge port, and the main control module is communicatively connected to the non-transparent bridge port, and the main control module is used to manage access rights between the non-transparent bridge ports, especially between the root associations connected to the non-transparent bridge ports.

[0008] Furthermore, the non-transparent bridge port includes a packet monitoring module, and the packet monitoring module is used to parse and obtain address information carried in the PCIe data packet.

[0009] Furthermore, the packet monitoring module includes addresses of resource mappings of other non-transparent bridge ports, wherein the resources include doorbell registers and / or temporary registers.

[0010] A second aspect of the present invention provides a communication management method for a PCIe switch as described above, wherein the root complexes connected to the PCIe switch can access each other, wherein a root complex connected to the PCIe switch can exchange information by performing a write operation on a doorbell register or a temporary register corresponding to another root complex connected to the PCIe switch, and the write operation comprises the steps of:

[0011] Query the resource usage of the target root complex:

[0012] If the resources corresponding to the target root complex are occupied, wait for a specified period of time and then requery until the resources corresponding to the target root complex are free; and

[0013] If the resources corresponding to the target root complex are not occupied, the information to be exchanged is written into the resources corresponding to the target root complex, wherein the resources include a doorbell register or a temporary register.

[0014] Furthermore, querying the resource usage corresponding to the target root complex includes:

[0015] The root complex sends a memory read packet to the non-transparent bridge port connected to it;

[0016] The non-transparent bridge port parses the memory read packet, and sends a permission request to the main control module according to a target address in the memory read packet;

[0017] The main control module queries the resource occupancy corresponding to the target root complex:

[0018] If the resources corresponding to the target root complex are occupied, returning a busy state to the non-transparent bridge port; and

[0019] If the resources of the target root complex are not occupied, recording the port for applying for permission and returning a success status to the non-transparent bridge port; and

[0020] The non-transparent bridge port composes a PCIe completion packet and sends it to the root complex.

[0021] Furthermore, when the main control module receives multiple permission application requests at the same time, it grants permission to the non-transparent bridge port with the highest priority according to a preset priority.

[0022] Furthermore, the non-transparent bridge port parses the memory read packet through a packet monitoring module and composes a PCIe completion packet, wherein the memory read packet and the PCIe completion packet are PCIe standard data packets.

[0023] Further, writing the information to be exchanged into the resource corresponding to the target root complex includes:

[0024] The root complex sends a memory write packet to the non-transparent bridge port connected to it;

[0025] The non-transparent bridge port parses the memory write packet and sends a write request to the main control module according to the address carried in the memory write packet; and

[0026] After receiving the write request, the main control module determines whether the write request port is consistent with the recorded port for which permission has been applied:

[0027] If they are consistent, the data is written to the resource corresponding to the target root complex and a success status is returned; and

[0028] If they are inconsistent, a failure status is returned.

[0029] Furthermore, the non-transparent bridge port parses the memory write packet through a packet monitoring module, wherein the memory write packet is a PCIe standard data packet.

[0030] Furthermore, the address carried by the memory write packet includes a mapping address of a resource corresponding to the target root complex at the current non-transparent bridge port.

[0031] Furthermore, the write operation further includes:

[0032] The root complex sends a memory read packet to obtain the write status, and after the write is successful, sends a release permission request to the main control module.

[0033] Furthermore, the permission release request is automatically sent after the root complex confirms that the writing is successful.

[0034] The present invention provides a PCIe switch with multiple non-transparent bridge ports and a communication management method thereof. First, a packet monitoring module is set at each non-transparent bridge port to implement the parsing of PCIe data packets. Then, a main control module is set to uniformly monitor and manage the resources corresponding to the root complex connected to each non-transparent bridge port, and uniformly manage the permission application of the non-transparent bridge port. It ensures the security of data interaction through a query feedback mechanism, and can effectively avoid the conflict problem of multiple root complexes accessing the same root complex resource at the same time. In addition, the permission application is basically implemented at the software level, so it can also simplify the hardware design of the PCIe switch, thereby saving design costs and improving the success rate of the design. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0036] Figure 1 A schematic diagram showing the structure of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention; and

[0037] Figure 2 A schematic flow chart of a write operation between root complexes of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that the various embodiments can be implemented without one or more specific details or with other replacement and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid blurring the inventive point of the present invention. Similarly, for the purpose of explanation, specific numbers and configurations are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details. In addition, it should be understood that the various embodiments shown in the drawings are illustrative representations and are not necessarily drawn in correct proportions.

[0039] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in this specification does not necessarily all refer to the same embodiment.

[0040] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment, rather than limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0041] In the present invention, each module according to the present invention can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented by a computer program flow, for example, the module can be implemented by a code segment (such as a code segment of a language such as C, C++, etc.) stored in a storage device (such as a hard disk, a memory, etc.), wherein when the code segment is executed by a processor, the corresponding function of the module can be implemented. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, the function of the module can be implemented by hardware programming a programmable device such as a field programmable gate array (FPGA), or the function of the module can be implemented by designing an application-specific integrated circuit (ASIC) including electronic devices such as a plurality of transistors, resistors and capacitors. When the module is implemented using firmware, the function of the module can be written into a read-only memory such as an EPROM or EEPROM of the device in the form of a program code, and when the program code is executed by a processor, the corresponding function of the module can be implemented.

[0042] In order to solve the conflict problem when multiple root complexes (RCs) access the resources of the same RC in a PCIe switch with multiple non-transparent bridging (NTB) ports, the present invention provides a PCIe switch with multiple non-transparent bridge ports and a communication management method thereof. The PCIe switch is connected to multiple RCs through multiple NTB ports, wherein the resources of each NTB port, such as the address of a doorbell register or a temporary register, are mapped to the address space (bar space) of other NTB ports. At the same time, a packet monitoring module is present on each NTB port to monitor the memory read packet (memory read packet) or memory write packet (memory write packet) of the PCIe standard, and the address in the memory read packet or memory write packet is parsed to determine whether it falls into the address space of the resources of other NTB ports. When it is detected that the address in the memory read packet or memory write packet falls within the range of a certain NTB port address space, the packet monitoring module will send a permission request to the main control module. The main control module contains the doorbell register or temporary register of each port, and at the same time detects and processes the conflicts caused by the requests of the packet monitoring module, and responds to the read and write requests of each packet monitoring module.

[0043] The scheme of the present invention is further described below in conjunction with the accompanying drawings of the embodiments.

[0044] Figure 1 FIG. 2 is a schematic diagram showing the structure of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention. Figure 1 As shown, a PCIe switch with multiple non-transparent bridge ports includes a main control module 101 and N non-transparent bridge ports 121 to 12N. As shown in the figure, the main control module 101 and the non-transparent bridge ports 121 to 12N are communicatively connected. Wherein N is a natural number greater than 1.

[0045] The non-transparent bridge ports 121 to 12N are used to isolate the address spaces of different PCIe bus domains to connect different RC 001 to 00N. In an embodiment of the present invention, the non-transparent bridge ports 121 to 12N can parse the PCIe standard data packets sent by the RC001 to 00N to obtain the address information carried therein. As mentioned above, in one embodiment of the present invention, the Bar space of each NTB port contains the mapping addresses of all other NTB port resources, so that it can determine the final sending target of the PCIe standard data packet according to the address information obtained by the analysis. Among them, the PCIe standard data packet includes a memory read packet and a memory write packet. In one embodiment of the present invention, the monitoring and analysis of the PCIe standard data packet by the NTB port is realized by a packet monitoring module, as shown in 1211 to 12N1 in the figure, each NTB port includes a packet monitoring module. The packet monitoring module is provided with a packet monitoring logic for parsing and obtaining the address information carried in the PCIe data packet. In one embodiment of the present invention, the packet monitoring module also extracts data from the PCIe data packet, and then sends the extracted data to the main control module when applying for permission. In one embodiment of the present invention, the packet monitoring logic is completely implemented at the hardware level. However, it should be understood that in other embodiments of the present invention, the packet monitoring module can also use software, firmware or a combination of hardware to implement this function.

[0046] The main control module 101 can not only manage the access rights between the non-transparent bridge ports, especially between the root complexes connected to the non-transparent bridge ports, but also the resources of the NTB ports, that is, the doorbell registers and / or temporary registers of each NTB port. This allows the main control module 101 to uniformly manage the resources corresponding to the RC and the permission applications of each NTB port, thereby solving the conflict problem of multiple RCs accessing the same resource.

[0047] Each RC connected to the PCIe switch can access each other's corresponding doorbell registers and temporary registers to exchange information. Generally speaking, when multiple RCs simultaneously access resources corresponding to the same RC and perform write operations, access conflicts are likely to occur. Based on this, Figure 2 FIG. 1 is a flow chart showing a write operation between root complexes of a PCIe switch with multiple non-transparent bridge ports according to an embodiment of the present invention. Figure 2 As shown, the write operation includes:

[0048] First, in step 201, apply for permission. When an RC wants to write to the doorbell register or temporary register corresponding to the target RC to send a message, it first needs to apply for permission, that is, query the resource occupancy corresponding to the target RC. If the resource is occupied, wait for a certain period of time and query again until the resource is free. If the resource is not occupied, the permission application is successful, and the process goes to step 202 to perform subsequent write operations. In one embodiment of the present invention, the RC can query the resource occupancy by sending a memory read packet and apply for permission. Specifically, it includes:

[0049] The RC sends a memory read packet to the NTB port connected thereto, wherein the memory read packet is a PCIe standard data packet;

[0050] The NTB port parses the memory read packet, for example, through its packet monitoring module, and sends a permission request to the main control module according to the target address in the memory read packet; and

[0051] The main control module inquires about the resource occupancy corresponding to the target RC and gives feedback. In one embodiment of the present invention, the feedback of the main control module includes two types: busy state and success state. When the main control module detects that the resource corresponding to the target RC is occupied, it returns a busy state to the NTB port, and when the resource is not occupied, it indicates that the request is successful, so it returns a success state. In one embodiment of the present invention, after the NTB port receives the feedback information sent by the main control module, it forms a PCIe completion packet (cpld packet) and returns it to the RC, and then the RC can perform the next step, such as continuing to send a memory read packet to query the resource occupancy, or sending a memory write packet to write information, etc. In actual applications, there may be a situation where multiple RCs apply for permissions at the same time. To solve this problem, in one embodiment of the present invention, when the main control module receives multiple permission application requests at the same time, it will grant permissions to the non-transparent bridge port with the highest priority according to the preset priority. In one embodiment of the present invention, the preset priority refers to sorting from large to small according to the port. In another embodiment of the present invention, the preset priority refers to sorting from small to large according to the port. In yet another embodiment of the present invention, the preset priority level may also be customized by the user according to actual needs;

[0052] In step 202, the port is recorded. When the main control module detects that the resource corresponding to the target RC is not occupied, the port is recorded for comparison when writing information later;

[0053] Next, in step 203, a memory write packet is sent. After the RC receives the PCIe completion packet and successfully applies for permission, it sends a memory write packet to the NTB port to write the information to be exchanged into the resources corresponding to the target root complex. In one embodiment of the present invention, the memory write packet is a PCIe standard data packet, and the address carried in the memory write packet is the address of the resource corresponding to the target RC mapped to the bar space of the NTB port connected to the RC, and the data in the memory write packet is the information to be exchanged;

[0054] Next, in step 204, a write request is generated. The NTB port parses the memory write packet through, for example, a packet monitoring module to obtain the address it carries, and determines whether the address falls within the address space of the resource corresponding to the target RC. If so, a write request is generated and sent to the main control module;

[0055] Next, in step 205, it is determined whether the ports are consistent. After receiving the write request, the main control module determines whether the write request port is consistent with the recorded port for which permission has been applied. If they are consistent, it proceeds to step 206 to complete the writing. If they are inconsistent, it returns to a failed state.

[0056] In step 206, the writing is completed. When the write request port is consistent with the recorded port of the applied permission request, the main control module writes the information to be exchanged into the resource corresponding to the target RC and returns a success status; and

[0057] Finally, in step 207, the permission is released. After the RC sends the memory write packet, it continues to send the memory read packet to obtain the write status. The packet monitoring module composes a PCIe completion packet (cpldpacket) and returns it to the RC to report the write status. If successful, a permission release request is sent to the main control module. The main control module releases the access rights of the NTB port to the target RC. At this time, other RCs can obtain the access rights to the target RC to perform subsequent operations. In one embodiment of the present invention, the RC automatically sends a permission release request after obtaining the successful write status and confirming that the write has been successful.

[0058] It can be seen that in the embodiments of the present invention, the application, use and release of permissions are all implemented using PCIe packages, which eliminates the need to set up additional management logic, and process control is more implemented at the software level, thereby making the hardware design simple and clear, saving the cost of chip design and increasing the success rate of chip design.

[0059] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should only be defined according to the attached claims and their equivalents.

Claims

1. A PCIe switch with multiple non-transparent bridge ports, characterized in that: include: A non-transparent bridge port, which is configured to connect to the RC and parse the PCIe data packet, the non-transparent bridge port includes a packet monitoring module, the packet monitoring module is configured to parse and obtain the address in the PCIe data packet, and the address space of the non-transparent bridge port includes the address of the resource mapping of other non-transparent bridge ports, wherein the resource includes a doorbell register and / or a temporary register; as well as A main control module includes a doorbell register and / or a temporary register of the non-transparent bridge port, and the main control module is communicatively connected to the non-transparent bridge port, and the main control module is configured to manage access rights between root complexes connected to the non-transparent bridge port.

2. The communication method of the PCIe switch according to claim 1, wherein: The root complex connected to the PCIe switch exchanges information by performing a write operation on resources corresponding to another root complex connected to the PCIe switch, wherein the write operation comprises the steps of: Query the resource usage of the target root complex: If the resources corresponding to the target root complex are occupied, wait for a specified period of time and then requery until the resources corresponding to the target root complex are free; and If the resources corresponding to the target root complex are not occupied, the information to be exchanged is written into the resources corresponding to the target root complex, wherein the resources include a doorbell register or a temporary register.

3. The communication method according to claim 2, characterized in that: Querying the resource usage of the target root complex includes the following steps: The root complex sends a memory read packet to the non-transparent bridge port connected to it; The non-transparent bridge port parses the memory read packet, and sends a permission request to the main control module according to a target address in the memory read packet; The main control module queries the resource occupancy corresponding to the target root complex: If the resources corresponding to the target root complex are occupied, returning a busy state to the non-transparent bridge port; as well as If the resources corresponding to the target root complex are not occupied, the port for applying for permission is recorded, and a success status is returned to the non-transparent bridge port; and The non-transparent bridge port composes a PCIe completion packet and sends it to the root complex.

4. The communication method according to claim 3, characterized in that: The non-transparent bridge port parses the memory read packet through a packet monitoring module and composes a PCIe completion packet, wherein the memory read packet and the PCIe completion packet are PCIe standard data packets.

5. The communication method according to claim 3, characterized in that: Also includes the steps: When the main control module receives multiple permission application requests at the same time, it grants permission to the non-transparent bridge port with the highest priority according to a preset priority.

6. The communication method according to claim 2, characterized in that: Writing the information to be exchanged into the resource corresponding to the target root complex comprises the steps of: The root complex sends a memory write packet to the non-transparent bridge port connected to it; The non-transparent bridge port parses the memory write packet and sends a write request to the main control module according to the address carried in the memory write packet; as well as After receiving the write request, the main control module determines whether the write request port is consistent with the recorded port for which permission has been applied: If they are consistent, the data is written to the resource corresponding to the target root complex and a success status is returned; and If they are inconsistent, a failure status is returned.

7. The communication method according to claim 6, characterized in that: The non-transparent bridge port parses the memory write packet through a packet monitoring module, wherein the memory write packet is a PCIe standard data packet.

8. The communication method according to claim 6, characterized in that: The address carried by the memory write packet includes a mapping address of a resource corresponding to the target root complex at the current non-transparent bridge port.

9. The communication method according to claim 2, characterized in that: The writing operation further comprises the steps of: The root complex sends a memory read packet to obtain the write status, and after the write is successful, sends a release permission request to the main control module.

Images