A PCIe switch and a method for multicasting data packets by multiple RCs in parallel

Abstract

This invention relates to the field of PCIe switch technology, specifically to a PCIe switch comprising a PCIe switch body, on which multiple USP ports, NTB ports, and DSP ports are configured. Each USP port and NTB port is configured with a multicast manager for multicast operations, and each USP port and NTB port is configured with a multicast request management module. The multicast request management module defines a three-bit register mapped in the bar space of the USP / NTB port. The PCIe switch of this invention supports connecting multiple RCs, and each RC connected to the USP / NTB port can send multicast data packets. Furthermore, it not only supports conformity checks on multicast data packets but also generates routing tables and sends them to the corresponding ports. It also solves the congestion problem caused by multiple RCs simultaneously sending multicast data packets.

Description

Technical Field

[0001] This invention relates to the field of PCIe switch technology, specifically to a PCIe switch and a method for parallel transmission of multicast data packets by multiple RCs. Background Technology

[0002] Peripheral Component Interconnect Express (PCIe) is a local bus technology and interface standard. A PCIe switch is a device in the PCIe standard that can expand PCIe interfaces and interconnect different devices with PCIe interfaces for information exchange and processing. PCIe devices in the PCIe bus domain to which the RC belongs can be accessed through the Root Complex (RC). Some PCIe switches support the PCIe Non-Transparent Bridging (NTB) function, which can isolate the address spaces of different PCIe bus domains to connect different RCs.

[0003] Multicast is a data transmission mode in PCIe switches that allows a device on one port of a PCIe switch to send data to multiple devices connected to the same PCIe switch port simultaneously through data replication. Compared to the traditional unicast mode, which involves sending and receiving data one at a time, multicast mode can reduce transmission operations and make efficient use of bandwidth, thereby significantly improving transmission efficiency.

[0004] Currently, existing PCIe switches typically only allow one RC to send multicast packets at a time. This is because of the send-receive characteristic of multicast. If multiple ports in a PCIe domain initiate multicast operations simultaneously, multiple sending ports may compete for sending rights, leading to congestion, a significant performance degradation, or even deadlock due to mutual waiting. Figure 1 The diagram shown illustrates the sending of multicast data packets by a PCIe switch in the prior art. When multiple RCs want to perform multicast operations to back up data, this cannot be achieved due to the aforementioned limitations.

[0005] To support simultaneous multicast operations on multiple ports, multiple PCIe switches would be required, leading to more complex system design and higher costs. Therefore, we propose a method for using PCIe switches and their multiple RCs to transmit multicast packets in parallel. Summary of the Invention

[0006] The purpose of this invention is to provide a PCIe switch and a method for parallel transmission of multicast data packets by multiple RCs, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A PCIe switch includes a PCIe switch body, which is configured with crossbar logic and multiple USP ports, NTB ports, and DSP ports. The USP ports and NTB ports are used to connect to RC, and the DSP ports are used to connect to PCIe terminal devices.

[0009] Both the USP port and the NTB port are configured with a multicast manager for multicast operations. The USP port sends multicast data packets and creates a routing table before sending them to each DSP port. The NTB port performs address translation on memory write packets sent to other PCIe domains. If the memory write packet after address translation hits the multicast window, a routing table is created for the multicast data packet, indicating the ports to which the multicast data packet arrives.

[0010] Both the USP port and the NTB port are equipped with a multicast request management module, which defines a three-bit register mapped in the bar space of the PCIe switch's USP / NTB port.

[0011] The first bit of the three-bit register is used to request the sending of multicast data packets. When it is written to 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports. The multicast request management modules of each port will send an MSI interrupt to that port to notify each RC.

[0012] The second bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When it is written to 1, it indicates that the RC on the current port is sending multicast packets and RCs on other ports need to wait. The multicast request management module of this port will send a busy flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party will send an MSI interrupt to the RC on this USP / NTB port to notify the RC.

[0013] The third bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When this bit is written to 1, it indicates that the RC on that port is idle or has changed from busy to idle, and agrees to allow RCs on other ports to perform multicast operations. The multicast request management module of that port will send an idle flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party records the number of packets received. If it is confirmed that all other RCs are in an idle state, the multicast request management module of that port will send an MSI interrupt to the USP / NTB port to notify the RCs.

[0014] Preferably, the scenarios for the MSI include: receiving a multicast request, receiving a busy response, and receiving an idle response.

[0015] Preferably, the crossbar logic is used to send multicast packets to each destination DSP port according to the routing table.

[0016] This invention also provides a method for parallel transmission of multicast data packets by multiple RCs, implemented through the aforementioned PCIe switch, comprising:

[0017] After the PCIe switch body is enumerated, each RC configures the multicast function register of its connection port according to the requirements and defines the multicast-related parameters of each port.

[0018] If the RC connected to the USP port needs to send multicast packets, the first bit of the multicast request management module configured in the RC will be sent to the multicast request management modules of other ports. After receiving the flag, the multicast request management modules of the other RCs will send an MSI to the port to notify the other RCs that there is an RC multicast request from the USP port.

[0019] Upon receiving a multicast request from an RC connected to the USP port, if other RCs are sending multicast data packets, the second bit of its own multicast request management module is configured. Its corresponding multicast request management module will send a busy flag to the multicast request management module corresponding to the requester. That is, the multicast request management module of the RC connected to the USP port will send an MSI notification to it, indicating that other ports are sending multicast data packets. The RC connected to the USP port needs to wait until it receives an MSI notification confirming the request.

[0020] Upon receiving a multicast request from the RC connected to the USP port, if the other RCs are not currently sending multicast packets or have completed sending multicast data packets after responding to the busy flag for a period of time, the other RCs will configure the third bit of the multicast request management module to indicate that they agree to allow the RC connected to the USP port to perform multicast operations. At this time, the multicast request management modules of the other RCs will send an idle flag to the multicast request management module of the RC connected to the USP port. The multicast request management module of the RC connected to the USP port will check whether all the other RCs have replied with the idle flag. If so, it will send an MSI to the RC connected to the USP port; otherwise, it will continue to wait for the busy other RCs to become idle.

[0021] After receiving an idle response from the multicast request management module, the RC connected to the USP port begins sending multicast data packets. If it receives a multicast request interruption from another RC during the sending process, it returns to busy; otherwise, the RC connected to the USP port continues to send multicast data packets until completion.

[0022] Preferably, the multicast-related parameters defined for each port include, but are not limited to, the multicast window size, the hit multicast group, and the multicast group's receive enable.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] The PCIe switch in this invention supports connecting multiple RCs, and each RC connected to the USP / NTB port can send multicast data packets;

[0025] The PCIe switch in this invention is equipped with a multicast manager, which not only supports checking multicast packets to conform to the PCIebase spec, but also generates routing tables and sends them to the corresponding ports.

[0026] The PCIe switch in this invention is equipped with a multicast request management module, which can solve the congestion problem caused by multiple RCs sending multicast data packets at the same time. Attached Figure Description

[0027] Figure 1 A schematic diagram illustrating the transmission of multicast data packets by a PCIe switch in the prior art;

[0028] Figure 2 This is a schematic diagram of a PCIe switch sending multicast data packets in this invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] A PCIe switch, such as Figure 2 The diagram illustrates the transmission of multicast data packets by a PCIe switch in this invention. It includes the PCIe switch body, which is configured with crossbar logic and multiple USP ports, NTB ports (e.g., NTB0, NTB1), and several DSP ports (e.g., DSP0, DSP1). The USP and NTB ports are used to connect to RCs (e.g., RC0, RC1, RC2), and the DSP ports are used to connect to PCIe endpoints or another switch. Figure 2 SSD in this context refers to SSD hard disk, which is one of the many PCIe terminal devices that can be connected to the DSP port. The illustration is just an example of an SSD.

[0031] Both the USP port and the NTB port are configured with a multicast manager for multicast operations. The USP port sends multicast data packets and creates a routing table before sending them to each DSP port. The NTB port performs address translation on memory write packets sent to other PCIe domains. If the memory write packet after address translation hits the multicast window, a routing table is created for the multicast data packet, indicating the ports to which the multicast data packet arrives.

[0032] Both the USP port and the NTB port are equipped with a multicast request management module, which defines a three-bit register mapped in the bar space of the PCIe switch's USP / NTB port.

[0033] The first bit of the three-bit register is used to request the sending of multicast data packets. When it is written to 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports. The multicast request management modules of each port will send an MSI interrupt to that port to notify each RC.

[0034] The second bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When it is written to 1, it indicates that the RC on the current port is sending multicast packets and RCs on other ports need to wait. The multicast request management module of this port will send a busy flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party will send an MSI interrupt to the RC on this USP / NTB port to notify the RC.

[0035] The third bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When this bit is written to 1, it indicates that the RC on that port is idle or has changed from busy to idle, and agrees to allow RCs on other ports to perform multicast operations. The multicast request management module of that port will send an idle flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party records the number of packets received. If it is confirmed that all other RCs are in an idle state, the multicast request management module of that port will send an MSI interrupt to the USP / NTB port to notify the RCs.

[0036] The scenarios for MSI include: receiving a multicast request, receiving a busy response, and receiving an idle response.

[0037] The crossbar logic is an important component of the switch. It is used to send data packets to various destination ports according to the routing table, and also to send multicast data packets to various destination ports according to the routing table. The bar space is the address space of the port.

[0038] This invention also provides a method for parallel transmission of multicast data packets by multiple RCs, implemented through the aforementioned PCIe switch, comprising:

[0039] After the PCIe switch body completes the enumeration, each RC configures the multicast function register of its connection port according to the requirements, and defines various multicast-related parameters for each port, such as the multicast window size, the hit multicast group, and the multicast group receive enable.

[0040] If an RC connected to the USP port needs to send multicast packets (RCs on multiple NTB ports need to send multicast packets in the same way; here we take an RC connected to the USP port sending multicast packets as an example), the RC configures the first step of the multicast request management module. The multicast request management module of this port will send a request flag to the multicast request management modules of other ports. After receiving the flag, the multicast request management modules of the other RCs will send an MSI to the port to notify the other RCs that there is an RC multicast request from the USP port.

[0041] Upon receiving a multicast request from an RC connected to the USP port, if other RCs are sending multicast data packets, the second bit of its own multicast request management module is configured. Its corresponding multicast request management module will send a busy flag to the multicast request management module corresponding to the requester. That is, the multicast request management module of the RC connected to the USP port will send an MSI notification to it, indicating that other ports are sending multicast data packets. The RC connected to the USP port needs to wait until it receives an MSI notification confirming the request.

[0042] Upon receiving a multicast request from the RC connected to the USP port, if the other RCs are not currently sending multicast packets or have completed sending multicast data packets after responding to the busy flag for a period of time, the other RCs will configure the third bit of the multicast request management module to indicate that they agree to allow the RC connected to the USP port to perform multicast operations. At this time, the multicast request management modules of the other RCs will send an idle flag to the multicast request management module of the RC connected to the USP port. The multicast request management module of the RC connected to the USP port will check whether all the other RCs have replied with the idle flag. If so, it will send an MSI to the RC connected to the USP port; otherwise, it will continue to wait for the busy other RCs to become idle.

[0043] After receiving an idle response from the multicast request management module, the RC connected to the USP port begins sending multicast data packets. If it receives a multicast request interruption from another RC during the sending process, it returns to busy; otherwise, the RC connected to the USP port continues to send multicast data packets until completion.

[0044] Preferably, the present invention notifies the RC via MSI. In other embodiments, the RC can also be notified via MSI-X. In addition, in other embodiments, the scenario in which the USP port and NT port on the switch of the present invention send multicast data packets simultaneously can be extended to the scenario of multiple PCIe switches being cascaded.

[0045] As can be seen from the above, the PCIe switch in this invention supports connecting multiple RCs, and each RC connected to the USP / NTB port can send multicast data packets. At the same time, the PCIe switch has a multicast manager, which not only supports checking multicast data packets to conform to the PCIe base spec, but also generates routing tables and sends them to the corresponding ports. Furthermore, the PCIe switch has a multicast request management module, which can solve the congestion problem caused by multiple RCs sending multicast data packets at the same time.

[0046] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A PCIe switch, comprising a PCIe switch body, wherein the PCIe switch body is configured with crossbar logic and multiple USP ports, NTB ports, and DSP ports, wherein the USP ports and NTB ports are used to connect to an RC (Remote Control Unit), and the DSP ports are used to connect to a PCIe terminal device; characterized in that: Both the USP port and the NTB port are configured with a multicast manager for multicast operations. The USP port sends multicast data packets and creates a routing table before sending them to each DSP port. The NTB port performs address translation on memory write packets sent to other PCIe domains. If the memory write packet after address translation hits the multicast window, a routing table is created for the multicast data packet, indicating the ports to which the multicast data packet arrives. Both the USP port and the NTB port are equipped with a multicast request management module, which defines a three-bit register mapped in the bar space of the USP / NTB port. The first bit of the three-bit register is used to request the sending of multicast data packets. When it is written to 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports. The multicast request management modules of each port will send an MSI interrupt to that port to notify each RC. The second bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When it is written to 1, it indicates that the RC on the current port is sending multicast packets and RCs on other ports need to wait. The multicast request management module of this port will send a busy flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party will send an MSI interrupt to the RC on this USP / NTB port to notify the RC. The third bit of the three-bit register is used to respond to requests from RCs on other ports to send multicast packets. When this bit is written to 1, it indicates that the RC on that port is idle or has changed from busy to idle, and agrees to allow RCs on other ports to perform multicast operations. The multicast request management module of that port will send an idle flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party records the number of packets received. If it is confirmed that all other RCs are in an idle state, the multicast request management module of that port will send an MSI interrupt to the USP / NTB port to notify the RCs.

2. The PCIe switch according to claim 1, characterized in that: The scenarios for MSI include: receiving a multicast request, receiving a busy response, and receiving an idle response.

3. The PCIe switch according to claim 1, characterized in that: The crossbar logic is used to send multicast packets to various destination ports according to the routing table.

4. A method for parallel transmission of multicast data packets by multiple RCs, implemented using a PCIe switch as described in any one of claims 1-3, characterized in that, include: After the PCIe switch body is enumerated, each RC configures the multicast function register of its connection port according to the requirements and defines the multicast-related parameters of each port. If the RC connected to the USP port needs to send multicast packets, the first step of the multicast request management module is configured in the RC. The multicast request management module of this port will send a request flag to the multicast request management modules of other ports. After receiving the flag, the multicast request management modules of the other RCs will send an MSI to the port to notify the other RCs that there is an RC multicast request from the USP port. Upon receiving a multicast request from an RC connected to the USP port, if other RCs are sending multicast data packets, the second bit of its own multicast request management module is configured. Its corresponding multicast request management module will send a busy flag to the multicast request management module corresponding to the requester. That is, the multicast request management module of the RC connected to the USP port will send an MSI notification to it, indicating that other ports are sending multicast data packets. The RC connected to the USP port needs to wait until it receives an MSI notification confirming the request. Upon receiving a multicast request from the RC connected to the USP port, if the other RCs are not currently sending multicast packets or have completed sending multicast data packets after responding to the busy flag for a period of time, the other RCs will configure the third bit of the multicast request management module to indicate that they agree to allow the RC connected to the USP port to perform multicast operations. At this time, the multicast request management modules of the other RCs will send an idle flag to the multicast request management module of the RC connected to the USP port. The multicast request management module of the RC connected to the USP port will check whether all the other RCs have replied with the idle flag. If so, it will send an MSI to the RC connected to the USP port; otherwise, it will continue to wait for the busy other RCs to become idle. After receiving an idle response from the multicast request management module, the RC connected to the USP port begins sending multicast data packets. If it receives a multicast request interruption from another RC during the sending process, it returns to busy; otherwise, the RC connected to the USP port continues to send multicast data packets until completion.

5. The method according to claim 4, characterized in that: The multicast-related parameters defined for each port include, but are not limited to, multicast window size, hit multicast groups, and multicast group receive enable.

Images