PCIe (Peripheral Component Interconnect Express) switch and multi-RC (Remote Controller) parallel multicast data packet sending method thereof

Abstract

The invention relates to the technical field of PCIe switches, in particular to a PCIe switch which comprises a PCIe switch body, and a plurality of USP ports, NTB ports and DSP ports are configured on the PCIe switch body. The USP port and the NTB port are respectively provided with a multicast manager used for carrying out multicast operation, the USP port and the NTB port are respectively provided with a multicast request management module, and the multicast request management module defines a three-bit register mapped in a bar space of the USP / NTB port. According to the invention, the PCIe switch supports the connection of a plurality of RCs, and each RC connected to a USP / NTB port can send a multicast data packet; meanwhile, checking conforming to regulations on the multicast data packet is supported, and a routing table can be generated and sent to a corresponding port; and the congestion problem caused by the fact that a plurality of RCs send multicast data packets at the same time can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of PCIe switch, in particular to a PCIe switch and a method for sending multicast data packets in parallel by multiple RCs. BACKGROUND

[0002] Peripheral Component Interconnect Express (PCIe) is a local bus technology and interface standard; PCIe switch is a device in PCIe standard, which can expand PCIe interface, interconnect different devices with PCIe interface to exchange and process information; through root complex (RC), PCIe devices in the PCIe bus domain to which the RC belongs can be accessed; some PCIe switches support PCIe Non-Transparent Bridging (NTB) function, thereby isolating the address spaces of different PCIe bus domains to connect different RCs.

[0003] Multicast operation is a data transmission mode in PCIe switch, which allows a port device in PCIe switch to send data to multiple devices connected to PCIe switch ports at the same time through data replication. Compared with the traditional unicast mode of one-to-one communication, multicast mode can reduce transmission operations, efficiently utilize bandwidth and greatly improve transmission efficiency.

[0004] Currently, existing PCIe switches can usually only send multicast data packets by one RC at a time, because of the characteristics of multicast one-to-many. If multiple ports of a PCIe domain simultaneously initiate multicast operations, congestion may occur due to simultaneous contention for sending rights by multiple sending ports, resulting in a significant reduction in performance, and even a deadlock scenario caused by mutual waiting. Figure 1 As shown in the prior art PCIe switch multicast data packet sending schematic diagram, when multiple RCs want to perform multicast operations and backup data, the above limitations cannot be implemented.

[0005] If multiple ports are to be supported for multicast operations at the same time, multiple PCIe switches will be considered, which brings more complex system design and higher cost. In view of this, a PCIe switch and a method for sending multicast data packets in parallel by multiple RCs are proposed. SUMMARY

[0006] The present application aims to provide a PCIe switch and a method for sending multicast data packets in parallel by multiple RCs to solve the problems raised in the background art.

[0007] To achieve the above object, the present application provides the following technical solutions: A PCIe switch comprises a PCIe switch body, wherein a crossbar logic and a plurality of USP ports, NTB ports and DSP ports are arranged on the PCIe switch body, the USP ports and the NTB ports are used to connect RCs, and the DSP ports are used to connect PCIE terminal devices. The USP ports and the NTB ports are each provided with a multicast manager used for multicast operation, wherein the USP ports normally send multicast data packets to each DSP port after issuing the multicast data packets and creating a routing table; and the NTB ports perform address conversion on memory write packets sent to other PCIe domains, and if the address-converted memory write packets hit a multicast window, a routing table of the multicast data packets is created to indicate each port reached by the multicast data packets. The USP ports and the NTB ports are each provided with a multicast request management module, and the multicast request management module defines a three-bit register mapped in a bar space of a PCIe switch USP / NTB port. The first bit of the three-bit register is used to apply for sending multicast data packets, and when the first bit is written as 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports, and the multicast request management modules of the ports send MSI interrupts to the RCs to notify the RCs. The second bit of the three-bit register is used to respond to the request of the RCs of other ports to send multicast data packets, and when the second bit is written as 1, it indicates that the RCs of the current port are sending multicast packets, and the RCs of other ports need to wait, the multicast request management module of the port sends a busy flag to the multicast request management modules of the request parties, and the multicast request management modules of the request parties send MSI interrupts to the RCs of the USP / NTB ports to notify the RCs. The third bit of the three-bit register is used to respond to the request of the RCs of other ports to send multicast packets, and when the third bit is written as 1, it indicates that the RCs of the port are idle or are converted from busy to idle, and agree to the RCs of other ports to perform multicast operation, the multicast request management module of the port sends an idle flag to the multicast request management modules of the request parties, the multicast request management modules of the request parties record the number of received idle flags, and if it is confirmed that all other RCs are in an idle state, the multicast request management module of the port sends an MSI interrupt to the RCs of the USP / NTB ports to notify the RCs.

[0008] Preferably, the MSI scenarios include receiving a multicast request, receiving a busy reply and receiving an idle reply.

[0009] Preferably, the crossbar logic is used to send multicast data packets to each destination DSP port according to a routing table.

[0010] The application also provides a method for multicasting data packets by multiple RCs in parallel, which is realized by the PCIe switch, comprising the following steps: After the enumeration is completed, the PCIe switch body configures the multicast function registers of the connection ports of each RC according to the requirements, and defines the parameters related to the multicast of each port. If the RC connected to the USP port needs to send a multicast data packet, the RC configures the first bit of the multicast request management module, and the multicast request management module of the port will send a request flag to the multicast request management modules of other ports. After receiving the multicast request from the RC connected to the USP port, if the remaining RCs are sending multicast data packets, the second bit of the multicast request management module of the RC is configured, and the corresponding multicast request management module will send a busy flag to the multicast request management module of the request side, that is, the multicast request management module of the RC connected to the USP port will send an MSI to the RC, prompting that other ports are sending multicast data packets, and the RC connected to the USP port needs to wait until it receives the MSI interrupt indicating that the request is confirmed. After receiving the multicast request from the RC connected to the USP port, if the remaining RCs are not sending multicast packets or the remaining RCs respond to the busy flag for a period of time and complete the sending of multicast data packets, the remaining RCs will configure the third bit of the multicast request management module, indicating that the RC connected to the USP port is allowed to perform multicast operation, at this time the multicast request management module of the remaining RCs will send an idle flag to the multicast request management module of the RC connected to the USP port, and the multicast request management module of the RC connected to the USP port will check whether all the remaining RCs have replied to the idle flag, if so, it will send an MSI to the RC connected to the USP port, otherwise it will continue to wait for the busy remaining RCs to be idle. After receiving the idle response of the multicast request management module, the RC connected to the USP port starts sending multicast data packets, and if it receives the multicast request interrupt of the remaining RCs during the sending process, it replies busy, otherwise the RC connected to the USP port continues to send multicast data packets until completion.

[0011] Preferably, the definition of the parameters related to the multicast of each port includes but is not limited to the window size of the multicast, the hit multicast group, and the reception enable of the multicast group.

[0012] Compared with the prior art, the application has the following beneficial effects: The PCIe switch in the application supports connecting multiple RCs, and each RC connected to the USP / NTB port can send multicast data packets. The PCIe switch in the application has a multicast manager, which not only supports checking multicast data packets in accordance with the PCIebase spec, but also generates a routing table and sends it to the corresponding port. The PCIe switch in the application has a multicast request management module, which can solve the congestion problem caused by multiple RCs simultaneously sending multicast data packets. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A schematic diagram of the PCIe switch in the prior art sending a multicast data packet; Figure 2 A schematic diagram of the PCIe switch in the application sending a multicast data packet. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the application.

[0015] A PCIe switch, as shown in the figure, includes a PCIe switch body, a crossbar logic, a plurality of USP ports, NTB ports, and a plurality of DSP ports. Figure 2 The PCIe switch in the application sends a multicast data packet, including a PCIe switch body, a crossbar logic, a plurality of USP ports, NTB ports, and a plurality of DSP ports. The USP ports and the NTB ports are used to connect RCs, and the DSP ports are used to connect PCIE endpoint devices or another switch. Figure 2 SSD in the figure refers to an SSD hard disk, which is one of a plurality of PCIE endpoint devices that can be connected to the DSP port. In the figure, only SSD is used as an example. The USP ports and the NTB ports are both configured with a multicast manager for multicast operation. After the USP ports send a multicast data packet and create a routing table, they are normally sent to each DSP port. The NTB ports perform address conversion on the memory write packet sent to other PCIe domains. If the address-converted memory write packet hits a multicast window, a routing table is created for the multicast data packet to indicate the ports reached by the multicast data packet. The USP ports and the NTB ports are both configured with a multicast request management module, which defines a three-bit register mapped in the PCIe switch USP / NTB port bar space. The first bit of the three-bit register is used for applying to send multicast data packets, and when being written as 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports, and the multicast request management modules of each port send MSI interrupts to the RCs to inform the RCs; The second bit of the three-bit register is used for responding to the request of the RCs of other ports to send multicast data packets, and when being written as 1, it indicates that the RC of the current port is sending multicast packets, and the RCs of other ports need to wait, and the multicast request management module of the port sends a busy flag to the multicast request management module of the requester, and the multicast request management module of the requester sends an MSI interrupt to the RC of the USP / NTB port to inform the RC; The third bit of the three-bit register is used for responding to the request of the RCs of other ports to send multicast packets, and when being written as 1, it indicates that the RC of the port is idle or changes from busy to idle, and agrees that the RCs of other ports perform multicast operation, and the multicast request management module of the port sends an idle flag to the multicast request management module of the requester, and the multicast request management module of the requester records the number of received, and if it is confirmed that all other RCs are in idle state, the multicast request management module of the port sends an MSI interrupt to the RC of the USP / NTB port to inform the RC.

[0016] The MSI scenarios include receiving a multicast request, receiving a busy reply, and receiving an idle reply.

[0017] The crossbar logic is an important component of the switch, which is used to send data packets to each destination port according to the routing table, and is also used to send multicast data packets to each destination port according to the routing table, and the bar space is the address space of the port.

[0018] The application also provides a method for multiple RCs to send multicast data packets in parallel, which is realized by the PCIe switch, and the method comprises the following steps: After the enumeration is completed, the PCIe switch body configures the multicast function registers of the connection ports of each RC according to the requirements, and defines each parameter related to multicast of each port, such as the window size of multicast, the hit multicast group, and the reception enable of the multicast group. If the RC connected to the USP port needs to send multicast data packets (the RCs on multiple NTB ports need to send multicast data packets follow the same method, and here the RC connected to the USP port is taken as an example), the RC configures the first bit of the multicast request management module, and the multicast request management module of the port sends a request flag to the multicast request management modules of other ports, and the multicast request management modules corresponding to the remaining RCs send MSI to the port after receiving the flag, to inform the remaining RCs that there is a multicast request from the RC of the USP port; After receiving the multicast request from the RC connected to the USP port, if the rest of the RCs are sending multicast data packets, the second bit of the multicast request management module of the RC is configured, and the corresponding multicast request management module sends a busy flag to the request side corresponding multicast request management module, that is, the multicast request management module of the RC connected to the USP port sends an MSI notification to it, prompting that other ports are sending multicast data packets, and the RC connected to the USP port needs to wait until the MSI interrupt of the request being confirmed is received; After receiving the multicast request from the RC connected to the USP port, if the rest of the RCs are not sending multicast packets or the rest of the RCs respond to the busy flag for a period of time and complete sending the multicast data packets, the rest of the RCs will configure the third bit of the multicast request management module, indicating that the RC connected to the USP port agrees to perform the multicast operation, at this time the multicast request management module of the rest of the RCs will send an idle flag to the multicast request management module of the RC connected to the USP port, and the multicast request management module of the RC connected to the USP port will check whether all the rest of the RCs have replied to the idle flag, if so, it will send an MSI to the RC connected to the USP port, otherwise it will continue to wait for the busy rest of the RC to be idle. After receiving the idle response of the multicast request management module, the RC connected to the USP port starts sending multicast data packets, and if it receives a multicast request interrupt from the rest of the RCs during the sending process, it replies busy, otherwise the RC connected to the USP port continues to send multicast data packets until completion.

[0019] Preferably, the application notifies the RC through MSI, and in other embodiments, the RC can also be notified through MSI-X. Meanwhile, in other embodiments, the scenario in which the USP port and the NT port on the switch of the application send multicast data packets at the same time can also be extended to the scenario of cascading multiple PCIe switches.

[0020] As can be seen from the above, the PCIe switch in the application supports connecting multiple RCs, and each RC connected to the USP / NTB port can send multicast data packets. Meanwhile, the PCIe switch is provided with a multicast manager, which not only supports checking the multicast data packets in accordance with the PCIe base spec, but also generates a routing table and sends it to the corresponding port. In addition, the PCIe switch is provided with a multicast request management module, which can solve the congestion problem caused by multiple RCs sending multicast data packets at the same time.

[0021] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A PCIe switch, comprising a PCIe switch body, the PCIe switch body being configured with crossbar logic and a plurality of USP ports, NTB ports, and DSP ports, wherein the USP ports and NTB ports are used to connect to RCs, and the DSP ports are used to connect to PCIe terminal devices; characterized in that: The USP port and the NTB port are both configured with a multicast manager for performing multicast operations. The USP port sends a multicast data packet and creates a routing table, which is then sent to each DSP port as normal. The NTB port performs address translation on memory write packets sent to other PCIe domains. If the memory write packet after address translation hits the multicast window, a routing table is created for the multicast data packet, indicating the ports to which the multicast data packet arrives. The USP port and the NTB port are both configured with a multicast request management module, and the multicast request management module defines a three-bit register mapped in the USP / NTB port bar space; The first bit of the three-bit register is used to request the sending of a multicast data packet. When it is written as 1, the multicast request management module sends a request flag to the multicast request management modules of other USP / NTB ports. The multicast request management module of each port will send an MSI interrupt to the port to notify each RC. The second bit of the three-bit register is used to respond to the request of the RC of other ports to send a multicast data packet. When it is written as 1, it means that the RC of the current port is sending a multicast packet and the RC of other ports need to wait. The multicast request management module of the port will send a busy flag to the multicast request management module of the requesting party, and the multicast request management module of the requesting party will send an MSI interrupt to the RC of the USP / NTB port to notify the RC; The third bit of the three-bit register is used to respond to the request of the RC of other ports to send multicast packets. When this bit is written to 1, it means that the RC of this port is idle or changes from busy to idle, and agrees to the RC of other ports to perform multicast operations. The multicast request management module of this port will send an idle flag to the multicast request management module of the requesting party. The multicast request management module of the requesting party records the number of received packets. If it is confirmed that all other RCs are in idle state, the multicast request management module of this port will send an MSI interrupt to the USP / NTB port to notify the RC.

2. The PCIe switch according to claim 1, wherein: The MSI scenarios include: receiving a multicast request, receiving a busy response, and receiving an idle response.

3. The PCIe switch according to claim 1, wherein: The crossbar logic is used to send the multicast data packets to each destination port according to the routing table.

4. A method for sending multicast data packets in parallel by multiple RCs, implemented by the PCIe switch according to any one of claims 1 to 3, characterized in that: include: After the PCIe switch is enumerated, each RC configures the multicast function register of its connected port as required, defining the multicast-related parameters of each port; If the RC connected to the USP port needs to send a multicast data packet, the RC configures the first bit of the multicast request management module. The multicast request management module of the port will send a request flag to the multicast request management modules of other ports. After receiving the flag, the multicast request management modules corresponding to the other RCs will send an MSI to the port to notify the other RCs that there is an RC multicast request from the USP port. After receiving a multicast request from the RC connected to the USP port, if other RCs are sending multicast data packets, the second bit of its own multicast request management module is configured, and its corresponding multicast request management module sends a busy flag to the multicast request management module corresponding to the requester, that is, the multicast request management module of the RC connected to the USP port will send an MSI notification to it, prompting that other ports are sending multicast data packets, and the RC connected to the USP port needs to wait until it receives the MSI interrupt indicating that the request is confirmed; After receiving the multicast request from the RC connected to the USP port, if the other RCs are not sending multicast packets or the other RCs complete sending multicast data packets after a period of time after responding to the busy mark, the other RCs will configure the third bit of the multicast request management module, indicating that they agree to the RC connected to the USP port to perform the multicast operation. At this time, the multicast request management module of the other RCs will send an idle flag to the multicast request management module of the RC connected to the USP port. The multicast request management module of the RC connected to the USP port will check whether all the other RCs have replied to the idle flag. If so, it will send MSI to the RC connected to the USP port, otherwise it will continue to wait for the busy other RCs to become idle. After the RC connected to the USP port receives the idle response from the multicast request management module, it starts sending multicast data packets. If it receives a multicast request interrupt from other RCs during the sending process, it replies busy. Otherwise, the RC connected to the USP port continues to send multicast data packets until completion.

5. The method according to claim 4, characterized in that: The multicast-related parameters defining each port include, but are not limited to, a multicast window size, a hit multicast group, and a receiving enable of the multicast group.

Images