Method for extending a PCIe system, PCIe switching device and PCIe system

Abstract

The application discloses a method for expanding a PCIe system, a PCIe switching device and a PCIe system. The PCIe switching device comprises an upstream port, a port agent and a plurality of first downstream ports. The upstream port is connected with the port agent, the port agent is connected with the plurality of first downstream ports, and the first downstream ports are used for connecting PCIe endpoint devices. The connections of the plurality of first downstream ports and the PCIe endpoint devices thereunder share the same bus number. By sharing the same bus number by the connections of the plurality of first downstream ports and the PCIe endpoint devices thereunder, the number of bus numbers occupied by the PCIe endpoint devices is reduced, and the capacity of the PCIe system is improved.

Description

[0001] This application claims priority to Chinese Patent Application No. 202010677632.7, filed on July 13, 2020, entitled "A Method for Improving PCIe Device Expansion Capability of a System", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, and particularly relates to a PCIe system expansion method, a PCIe switching device and a PCIe system. BACKGROUND

[0003] Peripheral Component Interconnect Express (PCIe) is a high-speed serial computer expansion bus standard, which is used to realize the connection between a processor and external devices, which are referred to as PCIe devices.

[0004] The PCIe protocol uses a 32-bit PCIe ID to identify PCIe devices in a system, which includes domain (16 bits): bus (8 bits): device (5 bits). function (3 bits)). The 8-bit bus identifier, 5-bit device identifier and 3-bit function identifier are referred to as BDF. The bus ID provides 256 bus numbers for each PCIe domain. Generally, a computer system only supports one PCIe domain, that is, the computer system only provides 256 bus numbers. Since the PCIe protocol is a point-to-point connection, the scale of PCIe devices that can be supported by a PCIe domain is restricted by the number of 256 bus numbers, and more PCIe devices cannot be supported in one PCIe domain. SUMMARY

[0005] The present application provides a PCIe system expansion method, a PCIe switching device and a PCIe system to solve the problem that the scale of PCIe devices is restricted by the number of 256 bus numbers in the prior art.

[0006] In a first aspect, the present application provides a PCIe switching device, which comprises an upstream port, a port agent and a plurality of first downstream ports, wherein the upstream port is connected to the port agent, the port agent is connected to the plurality of first downstream ports, and the first downstream ports are used to connect PCIe endpoint devices; the connections between the plurality of first downstream ports and the PCIe endpoint devices thereunder share the same bus number.

[0007] The port proxy device in the switching device of the first aspect is connected with a plurality of first downstream ports and provides proxy for the plurality of first downstream ports, and by sharing the same bus number for the connections between the plurality of first downstream ports and the PCIe endpoint devices thereunder, the number of bus numbers occupied by the PCIe endpoint devices is reduced, and the capacity of the PCIe system is improved.

[0008] In a possible implementation, the downstream bus number of the port proxy device is equal to the home bus number, wherein the home bus number is used to represent the largest bus number in the PCIe sub-tree under the port proxy device.

[0009] In another possible implementation, the PCIe switching device further comprises one or more second downstream ports, wherein the one or more second downstream ports are connected with the upstream port. The second downstream ports are connected to the upstream port and do not need to be provided with proxy services by the port proxy device.

[0010] In the enumeration process, the system enumeration software (for example, Basic Input Output System / Operating System, BIOS / OS) allocates BDFs to the PCIe devices (including bridge devices and endpoint devices) of the PCIe system. The bus numbers of the management topology of the bridge devices include a primary bus number, a secondary bus number and a subordinate bus number. After the enumeration is completed, the downstream bus number of the port proxy device is equal to the home bus number.

[0011] In another possible implementation, the downstream bus number of the port proxy device can be less than the home bus number, and the value of the home bus number minus the downstream bus number is less than the number of the first downstream ports. At this time, part of the plurality of first downstream ports share the same downstream bus number.

[0012] In another possible implementation, the port proxy device has the same home bus number as the plurality of first downstream ports. Specifically, the port proxy device sets the management topology bus numbers of the plurality of first downstream ports thereunder to be the same as the management topology bus number of itself.

[0013] In another possible implementation, the number of the first downstream port is used to represent a device identification of a PCIe endpoint device connected to the first downstream port. The processor can use the identification of the first downstream port to which the PCIe endpoint device is connected to distinguish each endpoint device under the port proxy device. The processor identifies the PCIe endpoint device based on the number of the first downstream port, in combination with a bus number D and a function number F.

[0014] In another possible implementation, the port proxy device is configured to receive a data packet sent by the PCIe endpoint device from the first downstream port, and replace a device identification in the data packet with the number of the first downstream port.

[0015] When the data packet sent by the PCIe endpoint device is a request TLP packet, the port proxy device is specifically configured to replace a device identification in a requester BDF carried by the request TLP packet with the number of the first downstream port to which the PCIe endpoint device is connected, or,

[0016] When the data packet sent by the PCIe endpoint device is a completion TLP packet, the port proxy device is specifically configured to replace a device identification in a completer BDF carried by the completion TLP packet with the number of the first downstream port to which the PCIe endpoint device is connected.

[0017] In another possible implementation, the port proxy device is configured to receive a data packet sent by the processor, use a value of a device identification in the data packet as a number of a destination port to which the data packet is received, change the device identification in the data packet to 0, and send the data packet to the destination port.

[0018] When the data packet sent by the processor is a configuration request TLP packet, replace a device identification in a destination BDF carried by the configuration request TLP packet with 0, or,

[0019] When the data packet sent by the processor is a completion TLP packet, replace a device identification in a requester BDF carried by the completion TLP packet with 0.

[0020] The configuration request TLP packet includes a configuration write request TLP packet and a configuration read request TLP packet.

[0021] In the embodiment, the processor replaces the device identifier of the PCIe endpoint device connected to the first downstream port with the number of the first downstream port in the enumeration process. In the prior art, the device identifier assigned to the PCIe endpoint device by the PCIe system is generally 0. In order to avoid affecting the PCIe endpoint device, the device identifier in the data packet sent to the PCIe endpoint device is changed to 0 by the port proxy device, so that the PCIe endpoint device does not need to change the software configuration for the scheme of the application.

[0022] In another possible embodiment, the configuration space register of the port proxy device is mapped with the configuration space registers of all the first downstream ports belonging to the port proxy device. Through the mapping of the first downstream port to the port proxy device by the configuration space register, the port proxy device can access the configuration space register of the first downstream port.

[0023] In another possible embodiment, the value of the enable register of the port proxy device is used to indicate whether the proxy function of the port proxy device is enabled. Through the enable register, the PCIe switching device can switch the proxy function, and when the proxy function is enabled, the scheme of the application is implemented, and when the proxy function is disabled, the PCIe switching device is used as a traditional PCIe switching device to provide services in the prior art.

[0024] Specifically, when the PCIe switching device is powered on or reset and initialized, the register of the port proxy device is set by firmware or reading a non-volatile memory (for example, an electrically erasable programmable read-only memory EEPROM).

[0025] In another possible embodiment, the port proxy device is configured to receive a first configuration request forwarded by the upstream port, and the first configuration request carries each bus number corresponding to the management topology of the port proxy device.

[0026] The port proxy device is further configured to set each bus number corresponding to the management topology of the port proxy device according to the first configuration request, and set the management topology of all the first downstream ports connected to the port proxy device to the same bus number.

[0027] By setting all the first downstream ports under the port proxy device to have the same management topology bus number as the port proxy device, the port proxy device proxies all the first downstream ports thereunder, and simulates all the first downstream ports thereunder as one proxy port to the upper system.

[0028] In another possible implementation, the port proxy device is configured to receive a second configuration request sent by the processor, the second configuration request carrying a bus number of a subordinate bus number of the port proxy device and carrying a device identifier m, where m is an integer greater than or equal to 0; and the port proxy device is further configured to route the second configuration request to a corresponding destination port according to the value of the device identifier m carried in the second configuration request, the number of the destination port being equal to m.

[0029] In this embodiment, the processor can enumerate all the first downstream ports under the port proxy device to the system by adjusting the value of m.

[0030] In another possible implementation, the PCIe switching device further includes one or more second downstream ports, and the port proxy device is configured to determine, according to a port enablement record, that the plurality of first downstream ports have enabled the proxy function and that the one or more second downstream ports have not enabled the proxy function. The port enablement record can be recorded in a bitmap register, and the bitmap value corresponding to each downstream port of the PCIe switching device recorded in the bitmap register can represent whether the endpoint device connected to each downstream port has enabled the proxy function. The port enablement record can also be a mapping table.

[0031] In another possible implementation, the address window of the port proxy device is the union of the address windows of all the first downstream ports connected thereto. When the port proxy device receives a packet in an address routing mode, the port proxy device can determine, by using the address window, whether the packet belongs to the address range of the first downstream ports connected thereto. When the address addressed by the processor belongs to the address window of the port proxy device, the port proxy device performs the addressing operation corresponding to the address.

[0032] In another possible implementation, the port proxy device is further configured to receive a message packet reported by the first downstream port connected thereto, replace the BDF of the first downstream port carried in the message packet with the BDF of the port proxy device, and route the message packet to the upstream port.

[0033] Further, the message packet includes an error packet. The error packet carries error information of the first downstream port.

[0034] When the message packet is an error packet, the port proxy device is further configured to further send state information of the first downstream port that reports the error packet.

[0035] The message packet further includes an interrupt packet.

[0036] In the embodiment, the port proxy device makes the first downstream port invisible to the upper layer system by modifying the BDF in the message packet sent by the first downstream port.

[0037] In another possible implementation, the sum of the number of bits of the device identifier and the number of bits of the function identifier of the PCIe endpoint device is 8.

[0038] In another possible implementation, the PCIe switch device is an Application Specific Integrated Circuit (ASIC) chip.

[0039] In another possible implementation, the PCIe switch device comprises a plurality of port proxy devices arranged in parallel, each of which is connected to a group of the first downstream ports. Each port proxy device provides proxy service for the first downstream ports in its group, and the downstream bus numbers of the first downstream ports in each group are different from those of the first downstream ports in other groups.

[0040] In a second aspect, the present application provides another PCIe switch device, comprising a processor, a memory, an upstream port and a plurality of first downstream ports, wherein the memory stores instructions which, when executed by the processor, implement the functions of the port proxy device of the first aspect.

[0041] In a third aspect, the present application provides a PCIe system, comprising a processor and the PCIe switch device of any one of claims 1-19.

[0042] In a possible implementation, the PCIe switch device comprises a first PCIe switch device and a second PCIe switch device, wherein the upstream port of the second PCIe device is connected to a second downstream port of the first switch device, and the second downstream port does not enable the proxy function.

[0043] In a fourth aspect, the present application provides another PCIe system, comprising a first PCIe switch device and a second PCIe switch device, wherein the first PCIe switch device is the PCIe switch device of the first aspect, and the second PCIe switch device is a conventional PCIe switch device, i.e. the second PCIe device does not comprise a port proxy device. The second PCIe switch device is an upstream or downstream device of the first PCIe switch device.

[0044] In a fifth aspect, the application provides a processor chip, comprising a root complex (RC) and the PCIe switch device as described in the above aspects, wherein an upstream port of the PCIe switch device is connected to a root port of the root complex, and the PCIe switch device is configured to implement the functions of the PCIe switch device in the above aspects.

[0045] In a sixth aspect, the application provides a method for extending a PCIe system, wherein the PCIe system comprises a processor and a PCIe switch device, and the PCIe switch device comprises an upstream port, a port proxy device and a plurality of first downstream ports, wherein,

[0046] the port proxy device is connected to the processor through the upstream port, the port proxy device is connected to the plurality of first downstream ports, and the first downstream ports are configured to connect PCIe endpoint devices,

[0047] the method comprises:

[0048] the processor sets the connections of the plurality of first downstream ports and the PCIe endpoint devices thereunder to the same bus number through the port proxy device.

[0049] Specifically, system enumeration software enumerates the PCIe endpoint devices in the PCIe system, and sets the bus numbers of the connections of the plurality of first downstream ports and the PCIe endpoint devices thereunder during the enumeration process.

[0050] In a seventh aspect, the application provides a computer program product, comprising a computer program stored in a computer readable storage medium, and the computer program is loaded by a processor to implement the functions of the first aspect or any possible implementation manner of the first aspect.

[0051] In an eighth aspect, the application provides a computer readable storage medium, configured to store a computer program, and the computer program is loaded by a processor to implement the functions of the above aspects or any possible implementation manner of the aspects. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a bus number allocation diagram in a PCIe system;

[0053] Figure 2 is a PCIe device management topology diagram corresponding to Figure 1

[0054] Figure 3 is a bus number allocation diagram in a PCIe system provided by an embodiment of the application;

[0055] Figure 4 ​is a PCIe device management topology diagram corresponding to Figure 3 is a PCIe device management topology diagram corresponding to

[0056] Figure 5 is a DPA configuration space diagram provided by an embodiment of the present application;

[0057] Figure 6 is another bus number allocation diagram in a PCIe system provided by an embodiment of the present application;

[0058] Figure 7 is a PCIe device management topology diagram corresponding to Figure 6 is a PCIe device management topology diagram corresponding to

[0059] Figure 8 is another bus number allocation diagram in a PCIe system provided by an embodiment of the present application;

[0060] Figure 9A and Figure 9B is a flow diagram of enumerating a PCIe device provided by an embodiment of the present application;

[0061] Figure 10 is an address window allocation diagram provided by an embodiment of the present application;

[0062] Figure 11 is a method flow diagram of ID routing provided by an embodiment of the present application;

[0063] Figure 12 is a device structure diagram of a PCIe switch device 1200 provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0065] The PCIe bus is a local bus of a processor system, and its function is similar to that of the PCI bus. The main purpose is to connect external devices in the processor system. The PCIe bus uses an end-to-end connection mode, and only one device can be connected at each end of a PCIe link. The two devices are data sending and receiving ends.

[0066] A typical PCIe system includes a root complex (RC), a PCIe switch, and a PCIe endpoint device. The RC can be located on a processor, and the RC itself occupies a bus number. The PCIe switch includes upstream ports and downstream ports, and the upstream ports are used to connect a root port (RP) of the RC or a downstream port of another PCIe switch. The root port of the RC, the upstream ports and the downstream ports of the PCIe switch can be bridge devices, which are used to connect other bridge devices or endpoint devices.

[0067] It should be noted that the PCIe switch can be a separate physical device, such as a physical switch or a switching chip, or a hardware functional module implemented in a chip, such as a hardware functional module integrated in a processor chip, which implements the function of the PCIe switch.

[0068] As shown in FIG. 1, an exemplary PCIe system includes a processor CPU, a root complex (RC), a PCIe switch, and a PCIe endpoint device. The RC can be located on the processor CPU, and the RC itself occupies a bus number. The PCIe switch includes upstream ports and downstream ports, and the upstream ports are used to connect a root port (RP) of the RC or a downstream port of another PCIe switch. The root port of the RC, the upstream ports and the downstream ports of the PCIe switch can be bridge devices, which are used to connect other bridge devices or endpoint devices. Figure 1 As shown in FIG. 1, an exemplary PCIe system includes a processor CPU, a root complex (RC), a PCIe switch, and a PCIe endpoint device. The RC can be located on the processor CPU, and the RC itself occupies a bus number. The PCIe switch includes upstream ports and downstream ports, and the upstream ports are used to connect a root port (RP) of the RC or a downstream port of another PCIe switch. The root port of the RC, the upstream ports and the downstream ports of the PCIe switch can be bridge devices, which are used to connect other bridge devices or endpoint devices. Figure 1The endpoint devices are labeled as EP_0, EP_1, …, EP_m. For example, the root port RP1 connected to the upstream port UP occupies bus X, and the link between RP1 and UP occupies bus N. At this time, the link between the PCIe switch internal upstream port and the m+1 downstream ports occupies bus N+1, and the links between the downstream ports and the corresponding endpoint devices occupy bus N+2, bus N+3, …, bus N+2+m, respectively. Here, X is an integer greater than or equal to 0, and N is an integer greater than X. For example, when the two RPs in the figure are located in different RCs, and the left RC where RP0 is located is enumerated first, the value of X depends on the bus numbers occupied by the PCIe structure under the left RC, for example, when the PCIe structure under the left RC occupies 5 bus numbers (i.e., buses 0-4 are occupied by the left RC and the PCIe structure under it), the value of X is 5, and the value of N is 6; when the two RPs in the figure are located in the same RC and the RC is the RC that the processor first enumerates, X is 0, and if the PCIe structure under the left RC occupies 5 bus numbers (i.e., buses 0-4 are occupied by the RC and the PCIe structure under RP0), the value of N is 5 at this time.

[0069] PCIe devices can be divided into two types, one is a bridge device, such as a root port RP, an upstream port, and a downstream port, and the other is an endpoint device (EP). At both ends of the PCI bridge, two buses are connected, which are the primary bus and the secondary bus. The PCIe bridge can manage the PCIe sub-tree under it. The bridge device has Subordinate Bus Number, Secondary Bus Number, and Primary Bus Number registers, the Primary Bus Number register stores the PCIe bus number upstream of the bridge device, the Secondary Bus Number register stores the PCIe bus number downstream of the bridge device, and the Subordinate Bus Number register stores the largest PCIe bus number in the PCIe sub-tree downstream of the bridge device, which is called the subordinate bus number.

[0070] As shown in Figure 2 , the PCIe bus number of the root port RP0 is stored in the Primary Bus Number register of the root port RP0, and the PCIe bus number of the root port RP1 is stored in the Primary Bus Number register of the root port RP1. The PCIe bus number of the root port RP0 is stored in the Subordinate Bus Number register of the root port RP1, and the PCIe bus number of the root port RP1 is stored in the Subordinate Bus Number register of the root port RP0. The PCIe bus number of the root port RP0 is stored in the Secondary Bus Number register of the root port RP0, and the PCIe bus number of the root port RP1 is stored in the Secondary Bus Number register of the root port RP1. Figure 1The corresponding PCIe device management topology diagram. The Primary Bus Number of RP1 is X, the BDF is X:0.0, the Secondary Bus Number is N, and the Subordinate Bus Number is N+2+m. The Primary Bus Number of the PCIe switch upstream port UP connected with RP1 is N, the BDF is N:0.0, the Secondary Bus Number is N+1, and the Subordinate Bus Number is N+2+m. The Primary Bus Number of the PCIe switch downstream port DP_0 is N+1, the BDF is N+1:0.0, the Secondary Bus Number and the Subordinate Bus Number are N+2. The Primary Bus Number of the PCIe switch downstream port DP_1 is N+1, the BDF is N+1:1.0, the Secondary Bus Number and the Subordinate Bus Number are N+3. The Primary Bus Number of the PCIe switch downstream port DP_m is N+1, the BDF is N+1:m.0, the Secondary Bus Number and the Subordinate Bus Number are N+2+m. The BDF of EP_0 connected with the downstream port DP_0 is N+2:0.0, the BDF of EP_1 connected with the downstream port DP_1 is N+3:0.0, and the BDF of EP_m connected with the downstream port DP_m is N+2+m:0.0. The aforementioned Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number can comply with the definition of the PCIe protocol, wherein the Primary Bus Number is used to represent the upstream bus number, the Secondary Bus Number is used to represent the downstream bus number, and the Secondary Bus Number is used to represent the largest bus number in the PCIe subtree hung under the current bus. The Secondary Bus Number and the Secondary Bus Number indicate the bus number range of the PCIe subtree under the current bus.

[0071] Figure 1 and Figure 2Only one possible bus allocation and management topology in PCIe system is given as an example to represent the bus number allocation and management topology. As can be seen, since PCIe is an end-to-end connection mode, one PCIe endpoint device needs to occupy one bus number, so that the number of PCIe endpoint devices is limited by the number of bus numbers.

[0072] As shown in Figure 3 , the embodiment of the present application provides a bus number allocation diagram in a PCIe system. Figure 3 As shown in the PCIe system includes a PCIe switch (i.e. a PCIe switching device). The embodiment of the present application adds a downstream port agent (DPA) in the PCIe switch, and the DPA provides proxy services for the ports downstream thereof, and each endpoint device connected to the downstream ports of the DPA shares the same bus, thereby saving the number of buses and providing the number of endpoint devices that the system can support. In the embodiment of the present application, the DPA is also referred to as a port proxy device, and in a specific implementation scenario, the DPA can be an ASIC chip or a software module. When the DPA is a software module, the DPA function is implemented by loading instructions by a processor.

[0073] In one possible implementation, the PCIe switch can be a separate hardware device, such as a switch or a switching chip; or a hardware function module integrated in other chips, which includes an upstream port, a port proxy device, and a plurality of downstream ports. For example, the PCIe switch can be integrated in a processor chip of a host.

[0074] Figure 3 As shown in the bus number allocation diagram in the PCIe system implements a bus number allocation logic different from the foregoing Figure 1 and Figure 2 . The PCIe system includes a processor CPU, a PCIe switching device (such as the PCIe switch shown in the figure), and a PCIe endpoint device. The subsequent embodiments of the present application take the PCIe switch as an example to describe the scheme.

[0075] There are a plurality of root ports (two RPs are taken as an example in the figure) on the processor, and the plurality of RPs can be located on one or more RCs (not shown in the figure). For example, RP0 can be connected to an endpoint device or a bridge device (not shown in the figure), and RP1 is connected to the upstream port UP of the PCIe switch. The PCIe switch includes m+1 downstream ports, which are marked as DP_0, DP_1, …, DP_m-1, DP_m, Figure 3 As shown in the figure, the endpoint devices are marked as EP_0, EP_1, …, EP_m-1, EP_m. As can be seen, the bus number of the PCIe switch is 0, and the bus number of the endpoint device EP_0 is 0.Figure 1 The difference is, Figure 3 The PCIe switch shown has a new DPA. The upstream port UP is connected to the DPA, and the DPA is connected to some or all of the downstream ports of the PCIe switch. The downstream ports connected to the DPA can be called downstream ports of the DPA, downstream ports subordinate to the DPA, or downstream ports with the proxy function enabled. For example, in the figure, DP_0, DP_1, ..., DP_m-1 are used as downstream ports with the proxy function enabled.

[0076] For example, the root port RP1 connected to the upstream port UP occupies bus X, and the connection between RP1 and UP occupies bus N. In this case, the connection from the upstream port UP to DPA inside the PCIe switch occupies bus N+1, the m-1 connections between downstream ports DP_0, DP_1, ..., DP_m-1 and EP_0, EP_1, ..., EP_m-1 share bus N+2, and the connection from UP to DP_m occupies bus N+3. Here, X is an integer greater than or equal to 0, and N is an integer greater than X. Examples of the values ​​of X and N are provided below. Figure 1 The same applies. For example, when the two RPs shown in the diagram are located in different RCs, and the left RC where RP0 is located is enumerated first, the value of X depends on the number of bus numbers occupied by the PCIe structure under the left RC. For example, when the PCIe structure under the left RRC occupies 5 bus numbers (i.e., buses 0-4 are occupied by the left RC and its PCIe structure), the value of X is 5, and the value of N is 6. When the two RPs shown in the diagram are located in the same RC and that RC is the RC that the processor enumerates first, the value of X is 0. If the PCIe structure under the left RC occupies 5 bus numbers (i.e., buses 0-4 are occupied), the value of N is 5.

[0077] like Figure 4 As shown, this application embodiment also provides a method with... Figure 3The corresponding PCIe device management topology diagram is shown below. RP1 has a Primary Bus Number of X, a BDF of X:0.0, a Secondary Bus Number of N, and a Subordinate Bus Number of N+3. The upstream port UP of the PCIe switch connected to RP1 has a Primary Bus Number of N, a BDF of N:0.0, a Secondary Bus Number of N+1, and a Subordinate Bus Number of N+3. The DPA in the PCIe switch has a Primary Bus Number of N+1, a BDF of N+1:0.0, a Secondary Bus Number of N+2, and a Subordinate Bus Number of N+2. Since DP_0, DP_1, ..., DP_m-1 have proxy functionality enabled, and the DPA provides proxy services for these m DPs, the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of these m DPs are the same as those of the DPA. The BDF of EP_0 is N+2:0.0, the BDF of EP_1 is N+2:1.0, ..., and the BDF of EP_m-1 is N+2:m-1.0. The proxy function of DP_m is not enabled, the BDF of DP_m is N+1:1.0, the Primary Bus Number of DP_m is N+1, the Secondary Bus Number is N+3, and the Subordinate Bus Number is N+3.

[0078] The foregoing Figure 3 and Figure 4 This application provides a bus allocation and management topology for a PCIe switch with DPA enabled, as illustrated in an embodiment. Specifically, DPA can proxy the functions of multiple downstream ports of the PCIe switch, presenting these multiple downstream ports as a single downstream port to the system. For example, one DPA can proxy the downstream ports of 32 PCIe switches.

[0079] When routing Transaction Layer Protocol (TLP) packets, the DPA replaces the 5-bit device ID in the BDF carried by the TLP packet with the downstream port number connected to the corresponding endpoint device before routing the TLP packet to the processor. Specifically, when the data packet sent by the PCIe endpoint device is a request TLP packet, the port proxy device is used to replace the device ID in the Requester BDF carried by the request TLP packet with the number of the first downstream port connected to the PCIe endpoint device; or, when the data packet sent by the PCIe endpoint device is a completion TLP packet, the port proxy device is used to replace the device ID in the Completer BDF carried by the completion TLP packet with the number of the first downstream port connected to the PCIe endpoint device. Through the above replacement operation, the DPA maps the PCIe endpoint devices connected to its downstream ports to the lower-level bus of the DPA. At this point, multiple PCIe endpoint devices connected to various downstream ports under DPA share the same bus number, thus solving the problem in the prior art where each endpoint device connected to a downstream port of a PCIe switch must be assigned a lower-level bus. This avoids the limitation that the number of endpoint devices in a PCIe system must be less than 256 (in actual product implementation, the total number of PCIe endpoints may be less than 248 because RC and other bridge devices will also occupy bus numbers). Furthermore, the embodiments of this application also avoid the waste problem caused by the PCIe devices connected to the downstream ports of a PCIe switch not fully utilizing all 256 PCIe IDs represented by device(5 bits).function(3 bits) in the prior art.

[0080] For example, in the prior art, a PCIe switch requires m+2 bus numbers to connect m PCIe endpoint devices. However, in this embodiment, when all m downstream ports of the PCIe switch have the proxy function enabled, the PCIe switch only requires 3 buses to connect m PCIe endpoint devices, thereby increasing the number of port devices supported by the PCIe system and improving bus utilization.

[0081] This application embodiment adds a DPA within the PCIe switch without requiring modifications to the existing BIOS, OS, PCIe subsystem, and PCIe device drivers.

[0082] The solutions disclosed in the embodiments of this application can greatly increase the number of endpoint devices that a PCIe can connect to. An example from the embodiments of this application is as follows: The method for enabling a computer system to support 7648 PCIe endpoint devices (such as solid-state drives) with only function 0 is as follows:

[0083] Eight root ports occupy one bus. Each root port is cascaded with a PCIe switch. Each PCIe switch has 32 DPAs. Each PCIe switch connects 32*32 PCIe endpoints with only function 0. n / 32 + 34*n = 256, where n is the number of PCIe switches. Each PCIe switch has 32 DPAs, and each DPA can proxy 32 downstream ports, requiring 34 buses. The calculated n = 7.52, rounded down to n = 7. Therefore, the maximum number of devices cascaded via PCIe switches is 7*32*32 = 7168, using 1 + 7*34 = 239 buses. The remaining 17 buses can be used to further expand the connection to (17-2)*32 = 480 devices, for a maximum of 7168 + 480 devices.

[0084] like Figure 5 As shown, exemplarily, this application provides a schematic diagram of the DPA configuration space in a PCIe switch, which differs from the DP configuration space defined by the PCIe protocol in that:

[0085] The DPA configuration space register BAR1 maps the configuration space registers of all DPs belonging to the DPA. The DPA driver can indirectly access the configuration space of all DPs under the DPA through the BAR1 register to handle PCIe switch DP links, errors, and hot-plugging.

[0086] The DPA configuration space adds a new DPA function structure, which includes an Enable register and a bitmap register. Specifically, bit 0 of the DPA Enable register is used to enable / disable the DPA function; for example, a value of 1 indicates that the DPA function is enabled. The DPA Port Bit Map LSB / MSB bitmap register implements a 32-bit bitmap (which can be expanded when the number of PCIe switch DPs exceeds 32), used to configure each DP that DPA needs to proxy. Each bit in the bitmap indicates whether the proxy function is enabled for the corresponding DP. LSB stands for Least Significant Bit, which refers to the least significant bit in binary, and MSB stands for Most Significant Bit, which refers to the most significant bit in binary. MSB is located at the leftmost and rightmost positions of the binary number.

[0087] For example, in combination Figure 3 In the downstream ports DP_0, DP_1, ..., DP_m, DP_0, DP_1, ..., DP_m-1 have the proxy function enabled, while DP_m has the proxy function disabled. At this time, the DPA bitmap register has more than or equal to m+1 bits. The bit in the bitmap used to represent DP_0, DP_1, ..., DP_m-1 can have a value of 1, and the bit in the bitmap used to represent DP_m can have a value of 0. A bit with a value of 1 indicates that the downstream port corresponding to that bit has the proxy function enabled, and a bit with a value of 0 indicates that the downstream port corresponding to that bit has the proxy function disabled.

[0088] The PCIe switch with DPA provided in this application embodiment can be applied to various PCIe architectures. For example, multiple PCIe switches with DPA can form a multi-layer architecture; PCIe switches with DPA can also be used in combination with traditional PCIe switches in a PCIe system; multiple DPAs can also be implemented in a single PCIe switch; thus, when the number of downstream ports that can be connected under a single DPA is limited, the number of endpoint devices that the PCIe system can support can be increased by using multiple DPAs in the PCIe switch.

[0089] For example, such as Figure 6 The diagram shown illustrates another PCIe system architecture provided in this application embodiment. In the illustrated PCIe system, two PCIe switches, including a DPA, are serially connected. The UP port of PCIe switch 1 is connected to the DP_m port of PCIeswitch 0. For bus number allocation details, please refer to [link to relevant documentation]. Figure 6 .

[0090] It should be noted that the aforementionedFigure 3 and Figure 6 In this implementation, all downstream ports under DPA share the same downstream bus number. In another possible implementation, some (not all) of the downstream ports under DPA share the same downstream bus number. In this case, the downstream bus number of the port proxy device is less than the home bus number. The value of the home bus number minus the downstream bus number is less than the aforementioned number of first downstream ports.

[0091] like Figure 7 As shown, this application provides an embodiment of the technology related to... Figure 6The corresponding PCIe device management topology diagram is shown below. RP1 has a Primary Bus Number of X, a BDF of X:0.0, a Secondary Bus Number of N, and a Subordinate Bus Number of N+6. The upstream port UP of PCIe switch 0 connected to RP1 has a Primary Bus Number of N, a BDF of N:0.0, a Secondary Bus Number of N+1, and a Subordinate Bus Number of N+6. The DPA in PCIe switch 0 has a Primary Bus Number of N+1, a BDF of N+1:0.0, a Secondary Bus Number of N+2, and a Subordinate Bus Number of N+2. Since DP_0, DP_1, ..., DP_m-1 have proxy functionality enabled, and DPA provides proxy services for these m DPs, the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of these m DPs are the same as those of DPA. The BDFs of the endpoint devices connected to DP_0, DP_1, ..., DP_m-1 of PCIe switch 0 are N+2:0.0, N+2:1.0, ..., N+2:m-1.0, respectively. DP_m of PCIe switch 0 has its proxy function disabled; its BDF is N+1:1.0, its Primary Bus Number is N+1, its Secondary Bus Number is N+3, and its Subordinate Bus Number is N+6. The UP of PCIe switch 1 is connected to DP_m of PCIe switch 0. The UP of PCIe switch 1 has a Primary Bus Number of N+3, a BDF of N+3:0.0, a Secondary Bus Number of N+4, and a Subordinate Bus Number of N+6. In PCIe switch 1, the DPA's Primary Bus Number is N+4, the BDF's is N+4:0.0, the Secondary Bus Number is N+5, and the Subordinate Bus Number is N+5.Because DP_0, DP_1, ..., DP_m-1 in PCIe switch 1 have proxy functionality enabled, and the DPA provides proxy services for these m DPs, the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of these m DPs in PCIe switch 1 are the same as those of the DPA in PCIe switch 1. The BDFs of the endpoint devices connected to DP_0, DP_1, ..., DP_m-1 in PCIe switch 1 are N+5:0.0, N+5:1.0, ..., N+5:m-1.0, respectively. DP_m in PCIe switch 1 does not have proxy functionality enabled; its BDF is N+4:1.0, and its Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number are all N+6. The BDF of the endpoint device connected to DP_m of PCIe switch 1 is N+6:0.0.

[0092] For example, such as Figure 8 The diagram shown illustrates another PCIe system architecture provided in this application embodiment. In this PCIe system, the PCIe switch contains n parallel DPAs, each DPA has m downstream ports with proxy functionality enabled, and the proxy functionality is disabled on the PCIe switch's DP_i. For bus number allocation details, please refer to [link to relevant documentation]. Figure 8 The content is as follows. The number of downstream ports under each DPA can be different, and whether the proxy function is enabled on the downstream ports can be flexibly configured.

[0093] It should be noted that in the aforementioned embodiments, the downstream port with the proxy function enabled can be referred to as the first downstream port, and the downstream port without the proxy function enabled can be referred to as the second downstream port.

[0094] The aforementioned schematic diagram of the PCIe system is only a possible implementation. It is understood that the PCIe switch provided in this application embodiment can be flexibly placed in various PCIe system structures, and the newly added DAP can be used to provide proxy functions for the downstream ports of the PCIe switch.

[0095] In conjunction with the foregoing embodiments, this application also provides a workflow for DPA in a PCIe system. The workflow includes at least one of the following: PCIe switch power-on or reset initialization; PCIe device enumeration; PCIe memory address resource allocation; TLP packet ID routing; TLP packet address routing; TLP packet implicit routing; endpoint device hot-plugging; error detection and handling, etc.

[0096] PCIe switch power-on or reset initialization:

[0097] During power-on / reset initialization, the PCIe switch uses firmware or electrically erasable programmable read-only memory (EEPROM) to configure bit 0 of the DPA Enable register and the Port Bit Map LSB / MSB of the DPA functional structure. For example, bit 0 of the DPA Enable register is set to 1, and the bits corresponding to the ports requiring DPA proxying in the bitmaps bits 0 to bit m-1 (for example, m is less than 31) of the DPA Port Bit Map LSB / MSB registers are set to 1, thereby enabling the DPA function and configuring the downstream ports (DP) that require DPA proxying.

[0098] like Figure 9A and 9B As shown, combined with Figure 3 The PCIe system architecture shown in this application provides a flowchart of enumerating PCIe devices, including:

[0099] 901: The processor enumerates the PCIe devices connected to bus 0, bus 1, ..., bus N-1 in sequence.

[0100] For example, this application embodiment assumes that the devices connected to bus 0, bus 1, ..., bus N-1 are all traditional PCIe devices, and the enumeration process follows the definition of the PCIe protocol. Specifically, the enumeration process can be executed by the processor running system enumeration software (BIOS and / or OS). This application embodiment does not limit the triggering of enumeration; in one possible implementation, it can follow the definition of the PCIe protocol.

[0101] 902: The processor enumerates the devices connected to bus N, writes the Primary Bus Number register of the upstream port of the PCIe switch to N, the Secondary Bus Number to N+1, and the Subordinate Bus Number to N+1.

[0102] For example, step 902 includes the following process:

[0103] The processor sends a configuration read request to the uplink port UP of the PCIe switch. This configuration read request is used to enumerate devices with bus N, device 0, and function 0, and obtain the Device ID and Vendor ID of that device. The Device ID and Vendor ID recorded in the PCIe device configuration space are used to record the device identifier and vendor identifier of the PCIe device. The device identifier in the configuration space is written by the vendor and is different from the device identifier in the BDF.

[0104] When the uplink port UP of the PCIe switch detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0, it returns the Device ID and Vendor ID register values ​​recorded in the upstream port configuration space to the processor. It should be noted that before the enumeration operation, the initial values ​​of the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of each port are all 0.

[0105] When the processor determines that the Device ID and Vendor ID values ​​returned by the upstream port of the PCIe switch are not 0xFFFF, it determines that a PCIe device exists on bus N. The processor sends a read request to configure the PCI Header Type register of the device with bus N, device 0, and function 0. It should be noted that the values ​​of Device ID and Vendor ID refer to the combined values ​​of Device ID and Vendor ID.

[0106] The uplink port UP of the PCIe switch receives a configuration read request for the PCI Header Type register from the processor. It detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0, and returns the value of the PCI Header Type register in the PCIe switch upstream port configuration space to the processor.

[0107] According to the PCIe protocol definition, the PCI Header Type register value of a PCI-PCI bridge is 10000001b. The processor, based on the PCIe protocol, identifies a device with bus N, device 0, and function 0 as a PCI-PCI bridge.

[0108] The processor sends a configuration read request to the PCI ExpressCapability register of the device configuration space, where bus is N, device is 0, and function is 0.

[0109] When the upstream port of the PCIe switch receives a configuration read request for the PCI Express Capability register, it detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0, and returns the value of the PCI Express Capability register in the upstream port configuration space of the PCIe switch to the processor.

[0110] When the processor reads that the PCI Express Capability register value returned by the upstream port of the PCIe switch is not 0xFFFF and the Device / Port Type field is 0101b, the processor recognizes the device with bus N, device 0, and function 0 as the upstream port of the PCIe switch according to the PCIe protocol.

[0111] The processor sends a configuration write request to the device with bus N, device 0, and function 0, which is used to write the Primary Bus Number register of the PCIeswitch upstream port to N, the Secondary Bus Number register to N+1, and the Subordinate Bus Number register to N+1.

[0112] After completing the enumeration of devices connected to bus N, the processor continues to enumerate devices connected to bus N+1. In this embodiment, the processor enumerates the DPAs in the PCIe switch.

[0113] Step 903: The processor sends a configuration read request to the uplink port UP of the PCIe switch. The configuration read request is used to enumerate the device with bus N+1, device 0, and function 0, and to obtain the device's Device ID and Vendor ID.

[0114] Step 904: Since the bus carried by the configuration read request in step 903 is N+1, after the PCIe switch upstream port determines that the Secondary Bus Number and Subordinate Bus Number of this port are N+1, it transfers the configuration read request to the internal routing and sends it to the DPA.

[0115] Step 905: The DPA detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0. The DPA further determines that the PCIeswitch DPA is enabled based on bit 0 of the configuration space enable register, and determines that there is a downstream port with the proxy function enabled based on the port enable record. The DPA returns the Device ID and Vendor ID register values ​​recorded in the port's configuration space to the processor.

[0116] It's important to note that the vendor ID recorded in the PCIe device configuration space is the vendor's ID, while the device ID is the vendor-provided ID for the PCIe device. The Device ID and Vendor ID are used to indicate the legitimacy of the PCIe device. In contrast, the Device ID in the BDF is assigned by the processor when enumerating PCIe devices and is used for managing PCIe devices. Therefore, the device ID in the BDF is different from the device ID recorded in the PCIe device configuration space.

[0117] The port enable record can be stored in a bitmap register. The bitmap values ​​corresponding to each downstream port of the PCIe switching device recorded in the bitmap register indicate whether the endpoint device connected to each downstream port has enabled the proxy function. The port enable record can also be a mapping table. The bitmap register can be a DPA Port BitMap LSB / MSB register.

[0118] Step 906: The processor receives the Device ID and Vendor ID returned by the DPA. If the values ​​of Device ID and Vendor ID are not 0xFFFF, it determines that a PCIe device exists on bus N+1. The processor sends a read request to configure the PCI Header Type register of the device with bus N+1, device 0, and function 0.

[0119] Step 907: The DPA receives a configuration read request for the PCI Header Type register from the processor, detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0, and returns the value of the PCI Header Type register in the DPA port configuration space to the processor.

[0120] According to the PCIe protocol definition, the PCI Header Type register value of a PCI-PCI bridge is 10000001b. The processor, based on the PCIe protocol, identifies a device with bus N+1, device 0, and function 0 as a PCI-PCI bridge.

[0121] Step 908: The processor sends a configuration read request to the PCIExpress Capability register of the device configuration space, where bus is N+1, device is 0, and function is 0.

[0122] Step 909: The DPA receives a configuration read request for the PCI Express Capability register, detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are all 0, and returns the value of the PCI Express Capability register in the DPA port configuration space to the processor.

[0123] When the processor reads that the PCI Express Capability register value returned by the DPA is not 0xFFFF and the Device / Port Type field is 0110b, the processor recognizes the device with bus N+1, device 0, and function 0 as the downstream port of the PCIe switch according to the PCIe protocol. That is, the processor recognizes the DPA as the downstream port of the PCIe switch.

[0124] Step 910: The processor sends a configuration write request to the device with bus N, device 0, and function 0, to write the Primary Bus Number of the PCIe upstream port to N, the Secondary Bus Number to N+1, and the Subordinate Bus Number to N+2.

[0125] Step 911: The processor sends a configuration write request to the device with bus N+1, device 0, and function 0, to write the DPA's Primary Bus Number register to N+1, Secondary Bus Number to N+2, and Subordinate Bus Number to N+2.

[0126] Step 912: The DPA receiver processor sends a configuration write request to the device with bus N+1, device 0, and function 0, writing the Primary Bus Number to N+1, the Secondary Bus Number to N+2, and the Subordinate Bus Number to N+2. Further, based on the value of the bitmap register, the DPA writes the Primary Bus Number of the downstream port with the DPA proxy function enabled to N+1, the Secondary Bus Number to N+2, and the Subordinate Bus Number to N+2. For example, with... Figure 3 Correspondingly, the Primary Bus Number of downstream ports DP_0, DP_1, ..., DP_m-1 is written as N+1, the Secondary Bus Number is N+2, and the Subordinate Bus Number is N+2.

[0127] The processor continues to enumerate the devices connected to bus N+2, that is, the processor enumerates the endpoint devices connected to downstream ports 0, 1, ..., m-1 under DPA.

[0128] Step 913: The processor sends a configuration read request, which is used to enumerate devices with bus N+2, device 0, and function 0, and to obtain the device ID and vendor ID of the device.

[0129] Step 914: Since the bus carried by the configuration read request in step 913 is N+1, after the PCIe switch upstream port determines that the Secondary Bus Number of this port is N+1 and the Subordinate Bus Number is N+2, it transfers the configuration read request to the internal route and sends it to the DPA.

[0130] Step 915: The DPA detects that the Primary Bus Number of this port is N+1, the Secondary Bus Number is N+2, and the Subordinate Bus Number is N+2. The DPA further determines that the PCIe switch DPA is enabled based on bit 0 of the configuration space enable register, and determines that there is a downstream port with enabled proxy functionality based on the values ​​of the LSB / MSB registers in the DPA Port Bit Map register. The DPA then routes the configuration read request to the downstream port DP_0.

[0131] Downstream port DP_0 detects that its Primary Bus Number is N+1, Secondary Bus Number is N+2, and Subordinate Bus Number is N+2, and routes the configuration read request to the endpoint device EP_0 connected to it.

[0132] Step 916: Endpoint device EP_0 returns a Completion TLP packet to downstream port DP_0. The Completion TLP packet carries the Device ID and Vendor ID register values ​​of endpoint device EP_0. Downstream port DP_0 routes the Completion TLP packet to DPA.

[0133] Step 917: The DPA receives the Complete TLP packet. The DPA determines that its function is enabled based on bit 0 of the enable register and that the downstream port DP_0 has its proxy function enabled based on the bitmap register. It then converts the device identifier in the Complete TLP packet to 0 and routes the Complete TLP packet to the processor through the upstream port of the PCIe switch. The device identifier in the Complete TLP message is carried in the Completer BDF, which is also called the Completer ID.

[0134] Step 918: Upon receiving the completion TLP packet, the processor determines that a PCIe device exists on bus N+2 if the values ​​of Device ID and Vendor ID in the completion TLP packet are not 0xFFFF. The processor then sends a read request to configure the PCI Header Type register of the device with bus N+2, device 0, and function 0.

[0135] Step 919: Similar to the previous scheme, PCIe endpoint device EP_0 returns the PCI Header Type register value of this device. The processor reads the PCI Header Type register value returned by PCIe endpoint device EP_0 as 00000000b. The system enumeration software identifies, according to the PCIe protocol, that the device with bus N+2, device 0, and function 0 is an endpoint device, and contains only 1 function.

[0136] Step 920: Similarly, the processor continues to read the PCI Express Capability register of the device configuration space where bus is N+2, device is 0, and function is 0.

[0137] Step 921: PCIe endpoint device EP_0 returns the PCI Express Capability register value. The system enumeration software reads that the PCIe endpoint device EP_0 returns a PCI Express Capability register value that is not 0xFFFF, and the Device / Port Type field is 0000b. Based on the PCIe protocol, the system enumeration software identifies the device with bus N+2, device 0, and function 0 as the endpoint device.

[0138] If an endpoint device contains multiple functions, the system enumeration software reads the PCI Header Type register value returned by PCIe endpoint device EP_0 as 10000000b. Based on the PCIe protocol, the system enumeration software identifies the device with bus N+2, device 0, and function 0 as an endpoint device, and it contains only multiple functions. The system enumeration software then repeats the above steps to enumerate endpoint devices with bus N+2, device 0, and functions 1 to 7 into the system.

[0139] After the processor (system enumeration software) completes the enumeration of downstream port DP_0, it needs to scan all device functions under bus N+2. In one specific implementation, a forced scan startup parameter is added to the computer operating system kernel. For example, for a Linux system, the startup parameter pci = pcie_scan_all can be added to the Linux kernel, allowing the processor to forcibly enumerate all device functions under bus N+2, i.e., perform a total of 256 scans. When the processor receives the PCI Header Type register value of 00000000b returned by EP_0, it avoids the processor mistakenly believing that the bus scan of bus N+2 has been completed because it finds that EP_0 is a device with only one function, thus missing the scan of other device functions on the bus N+2. For example, other parameter settings can also be used to enable the processor to perform a complete scan of devices connected to the lower-level bus of DPA, which will not be elaborated in the embodiments of this application.

[0140] With the above settings, after the processor completes the enumeration of devices with bus N+2 and device 0, in step 922, the processor continues to enumerate the functions of devices with bus N+2, device numbers 1 to m-1, and function numbers 1 to 7, where m is an integer greater than or equal to 1. The specific enumeration process is the same as the processor's enumeration process for devices with bus N+2, device 0, and function 0. This embodiment will not be described in detail here.

[0141] It should be noted that when a downstream port with the proxy function enabled is not connected to a PCIe endpoint device, the processor skips that port during the enumeration process, and also skips the device ID corresponding to that port, and continues to enumerate other downstream ports.

[0142] After the processor completes enumeration of all endpoint devices under bus N+2, the processor continues enumeration, traversing bus N+1.

[0143] Step 923: The processor sends a configuration read request for a device with bus N+1, device m, and function 0. The configuration read request is used to obtain the Device ID and Vendor ID of the device.

[0144] For example, Figure 3 In the PCIe architecture diagram shown, after enumerating the endpoint devices connected to downstream ports DP_0 to DP_m-1 under DPA, the processor continues to enumerate the devices connected to downstream port DP_m. At this point, the device ID in the configuration read request is 1 (i.e., for the application). Figure 3In the architecture, during the DP_m enumeration process, m is set to 1 in the following steps. It should be noted that when multiple downstream ports have the proxy function disabled, the device ID of the configuration read request is incremented by 1 for each port.

[0145] Step 924: Since the bus carried by the configuration read request in step 923 is N+1, after the PCIe switch upstream port determines that the Secondary Bus Number of this port is N+1 and the Subordinate Bus Number is N+2, it transfers the configuration read request to the internal route and sends it to DP_m.

[0146] Step 925: The downstream port DP_m of the PCIe switch detects that the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of this port are 0, and returns the Device ID and Vendor ID register values ​​of the downstream port DP_m to the processor.

[0147] The processor determines that the Device ID and Vendor ID returned by the downstream port DP_m of the PCIe switch are not 0xFFFF, and therefore determines that a device with bus N+1, device m, and function 0 exists.

[0148] Step 926: Similarly, the processor continues to read the PCI Header Type register of the device configuration space where bus is N+1, device is m, and function is 0. The processor obtains the PCI Header Type register value returned by the downstream port DP_m as 10000001b. The system enumeration software identifies the device with bus N+1, device is m, and function is 0 as a PCI-PCI bridge according to the PCIe protocol.

[0149] Step 927: Similarly, the processor continues to read the PCI Express Capability register of the device configuration space where bus is N+1, device is m, and function is 0. The processor reads that the PCIExpress Capability register value returned by the downstream port DP_m of the PCIe switch is not 0xFFFF, and the Device / Port Type field is 0110b. The system enumeration software identifies the device with bus N+1, device is m, and function is 0 as the downstream port of the PCIe switch according to the PCIe protocol.

[0150] Step 928: The processor configures the Primary Bus Number register to N, the Secondary Bus Number to N+1, and the Subordinate Bus Number to N+3 for the device with bus N, device 0, and function 0. That is, the Primary Bus Number register of the PCIeswitch upstream port is set to N, the Secondary Bus Number to N+1, and the Subordinate Bus Number to N+3.

[0151] Step 929: The processor configures the Primary Bus Number register to N+1, the Secondary Bus Number to N+3, and the Subordinate Bus Number to N+3 for the device with bus N+1, device m, and function 0. That is, the Primary Bus Number register of the downstream port DP_m of the PCIe switch is set to N+1, the Secondary Bus Number to N+3, and the Subordinate Bus Number to N+3.

[0152] Through steps 924-929, the processor completes the enumeration of DP_m. Further, the processor continues to enumerate bus N+3 and enumerates EP_m.

[0153] When there are other downstream ports where the proxy function is not enabled, the processor continues to enumerate other downstream ports and PCIe devices connected to other downstream ports in the same way.

[0154] In the above manner, the processor completed... Figure 3 The enumeration of PCIe systems is shown. In other possible implementations, when there are more lower-level PCIe switches in the PCIe system and more ports on the PCIe switches, the aforementioned steps can be repeated to complete the device enumeration of various PCIe systems.

[0155] PCIe memory address resource allocation:

[0156] The processor (running system enumeration software) allocates memory addresses to each PCIe device in the PCIe system. In one possible implementation, the processor can allocate memory addresses according to a conventional PCIe device memory address resource allocation algorithm.

[0157] The processor first iterates through the memory address resources declared by each PCIe endpoint device, then allocates the corresponding address space from the system memory address resources, writes the starting address to the BAR register of the PCIe endpoint device, and adjusts the address windows of each bridge device on the path from that endpoint device to the RP. For example, the Memory Limit register and MemoryBase register of the PCI-PCI bridge represent 32-bit address windows, and the Prefetchable Memory Limit register, Prefetchable Memory Base register, Prefetchable Memory Upper Base Address register, and Prefetchable Memory Upper Limit Address register represent 64-bit address windows.

[0158] Specifically, the PCIe switch DPA records the configuration write TLP packets for the Memory Base registers of PCIe endpoint devices from the system enumeration software when routing them to downstream ports of the PCIe switch. The PCIe switch can detect the 32-bit / 64-bit start addresses of all endpoint devices connected to the downstream port. The bridge device's address window (based on the 32-bit and 64-bit address windows of the PCI-PCI bridge) must include the address range used by all downstream PCIe devices. Therefore, the PCIe switch firmware (FW) can calculate the end address based on the start address of each downstream PCIe endpoint device and set it in the downstream port's Memory Limit register and Memory Base register (Prefetchable Memory Limit register, Prefetchable Memory Base register, Prefetchable Memory Upper Base Address register, and Prefetchable Memory Upper Limit Address register). For example, as shown... Figure 10 The diagram shown is a memory address resource allocation diagram provided in an embodiment of this application. In the PCIe system architecture, the address window of an upstream device contains the address windows of its downstream devices at each layer. Figure 10In the example, the address windows are contiguous. Specifically, the address window of the upstream port of the PCIe switch is [A,Z], the address window of the DPA in the PCIe switch is [A,Y], and the downstream ports DP_0, DP_1, ..., DP_m-1 in the PCIe switch have proxy functionality enabled. In this case, the address window of DP_0 proxies by the PCIe switch DPA is [A,B], the address window of DP_1 is [B,C], ..., the address window of DP_m-1 is [X,Y], and the address window of DP_m, which does not have proxy functionality enabled, is [Y,Z]. In one possible implementation, when there are enough registers to record discrete address windows, the address windows of each device can also use non-contiguous address spaces.

[0159] The PCIe protocol defines workflows such as ID-based routing, address-based routing, and implicit routing. When the PCIe switch with DPA provided in this application is applied in a PCIe architecture, the aforementioned workflows also need to be adjusted accordingly.

[0160] like Figure 11 The diagram shown is a flowchart illustrating an ID routing method according to an embodiment of this application. ID routing is generally used in the transmission flow of configuration requests (configuration read requests / write requests). The method includes:

[0161] Step 1101: The upstream port of the PCIe switch receives a configuration request from the processor, determines whether the bus number in the BDF carried in the configuration request falls within the range of the downstream bus and slave bus of the upstream port, and if so, transfers the configuration request to the internal routing of the PCIe switch and sends it to the DPA.

[0162] The configuration request can be a configuration read / write request (TLP) packet issued by the processor. It should be noted that this embodiment uses the example of a configuration request being sent to a DPA for illustration. When the upstream port needs to route the configuration request to a downstream port where the proxy function is not enabled based on the BDF carried in the configuration request, it can execute the process specified in the PCIe protocol.

[0163] Step 1102: The DPA receives the configuration request and determines whether the bus number in the target BDF carried in the configuration request falls within the range of the DPA's subordinate bus and slave bus. If so, the DPA routes the configuration request to the corresponding downstream port according to the D (i.e., device identifier, which can be represented by device (5 bits)) in the target BDF carried in the configuration request. The value of the device identifier carried in the configuration request is used as the number of the downstream port. The DPA changes the device identifier in the target BDF carried in the configuration request to 0.

[0164] Step 1103: The downstream port will route the configuration request to the endpoint device connected to that downstream port.

[0165] For specific examples of the configuration request processing in steps 1101-1103, please refer to steps 913 to 916.

[0166] Step 1104: If the bus number in the BDF carried in the DPA configuration request does not belong to the range defined by the lower-level bus and subordinate bus of the DPA, it indicates that a routing error has occurred. In this case, the DPA can refer to the error handling procedure for bridge devices defined in the protocol, and this embodiment of the application does not limit it.

[0167] Step 1105: The upstream port of the PCIe switch receives the completion TLP packet sent by the processor, and determines whether the bus number in the requester BDF carried in the completion TLP packet falls within the range of the downstream bus and subordinate bus of the upstream port. If so, the completion TLP packet is transferred to the internal routing of the PCIe switch and sent to the DPA.

[0168] Step 1106: DPA receives the completed TLP packet and determines whether the bus number in the requester BDF carried by the completed TLP packet falls within the range of DPA's subordinate and dependent buses. If so, DPA changes the D (i.e., device identifier, which can be represented by device (5 bits)) in the requester BDF carried by the completed TLP packet to 0, and then routes the completed TLP packet to the corresponding downstream port, where the value of the original device identifier in the requester BDF carried by the completed TLP packet is used as the number of the downstream port.

[0169] Step 1107: The downstream port routes the completed TLP packet to the endpoint device connected to the downstream port.

[0170] Step 1108: When DPA determines that the bus number in the requester BDF carried in the completed TLP packet does not fall within the range of DPA's subordinate buses and dependent buses, it indicates that a routing error has occurred and performs error handling operations.

[0171] Step 1109: The downstream port of the PCIe switch receives a completion TLP packet from the endpoint device and sends the completion TLP packet to the DPA.

[0172] Step 1110: The DPA receives the Complete TLP packet, determines the number of the downstream port that sent the Complete TLP packet, determines that the DPA function is enabled based on bit 0 of the enable register, and determines that the downstream port that sent the Complete TLP packet has enabled the proxy function based on the value of the bitmap register. Then, the DPA changes the device identifier in the Completer BDF carried in the Complete TLP packet to the number of the downstream port. The DPA then routes the Complete TLP packet to the processor through the upstream port.

[0173] Step 1111: After receiving the Complete TLP packet, if the DPA determines that the DPA function is not enabled based on bit 0 of the enable register, or determines that the downstream port sending the Complete TLP packet has not enabled the proxy function based on the value of the bitmap register, then the DPA does not modify the device identifier of the Complete TLP packet, but directly sends the Complete TLP packet to the processor through the upstream port.

[0174] The logic for address routing in a PCIe switch that includes DPA includes:

[0175] When the upstream port of a PCIe switch receives a memory operation request (e.g., a memory read / write request (TLP) packet) from the processor, it determines whether the destination address falls within the address window of the upstream port. If the destination address falls within the address window, the request is routed internally by the PCIe switch. The internal routing then determines whether the destination address falls within the address window of the PCIe switch's Data Access Provider (DPA). If the destination address falls within the DPA's address window, the upstream port routes the received memory operation request to the DPA. The DPA then determines whether the destination address falls within the address window of the downstream port it proxies. If the destination address falls within the address window of the downstream port proxies by the DPA, the memory operation request is routed to the downstream port proxies by the DPA. The downstream port then routes the memory operation request to the PCIe endpoint device it is connected to. If the DPA determines that the destination address falls within the address window of a downstream port not proxies by the DPA, a routing error has occurred.

[0176] When a PCIe downstream port receives a memory operation request (e.g., a memory read / write request (TLP) packet) from an endpoint device, it forwards the memory operation request to the DPA. The DPA checks whether the downstream port sending the memory operation request has proxy functionality enabled. If not, the DPA routes the memory operation request to the processor via the upstream port. If it has enabled, the DPA changes the device identifier in the memory operation request to the number of the downstream port that sent the request, and then routes the memory operation request to the processor via the upstream port. Specifically, the device identifier of the memory operation request is carried in the BDF (Browser Validation Function) within the request.

[0177] The logic for implicit routing in a PCIe switch that includes DPA includes:

[0178] When the upstream port of the PCIe switch receives an implicit TLP packet (i.e., a message packet) from the processor, it routes the data according to the message routing subfield in the implicit TLP packet.

[0179] When DPA receives an implicit TLP packet sent by an upstream port, it determines whether the downstream port that sent the implicit TLP packet has enabled the proxy function. If it has, it changes the device identifier in the requester BDF in the implicit TLP packet to 0. If not, it directly routes according to the message routing subfield in the TLP packet.

[0180] When an implicit TLP packet initiated by a downstream port proxied by a PCIe switch DPA passes through the PCIe switch DPA, the DPA needs to change the requester ID / BDF in the TLP packet to the BDF of the PCIe switch DPA.

[0181] Implicit TLP packets sent by downstream ports that are not proxied by the PCIe switch DPA are routed directly based on the message routing subfield in the TLP packet.

[0182] When a hot-plugging of an endpoint device occurs on a downstream port proxies by DPA, the interrupt message reported by the downstream port passes through DPA. DPA then modifies the BDF according to the aforementioned implicit routing rules, reports the interrupt to the processor, and triggers a DPA hot-plugging interrupt. DPA performs a port scan, similar to the forced device function scan described in the previous embodiment. In this case, DPA needs to forcibly scan the 256 device functions represented by device(5 bits).function(3 bits).

[0183] When a downstream port of the DPA agent reports an error message, the DPA changes the BDF of the error message to its own BDF and records the port's status information in the configuration space registers of both the DPA and the downstream port. When the processor receives the error message, it iterates through the configuration space registers of each downstream port under the DPA, determines the number of the downstream port that detected the error based on the previously recorded port status, and then performs error handling for that downstream port. Specifically, the configuration space register that records the port status can be the AER (Advanced Error Reporting) function register.

[0184] It should be noted that in the various embodiments of this application, there may be one or more bridge devices between the processor and the PCIe switch, and the processor communicates with the PCIe switch through the bridge devices. For simplicity, the above embodiments of this application are described using the example of a PCIe switch directly connected to the processor RP. It is understood that when there is a traditional PCIe bridge device between the processor and the PCIe switch, the intermediate PCIe bridge device can handle the communication between the processor and the PCIe switch provided in the embodiments of this application according to the definition of the PCIe protocol.

[0185] To ensure compatibility with current PCIe protocol scenarios, this embodiment only allows the DPA to map the PCIe endpoint devices connected to its proxied downstream ports to the device identifier in the BDF. Since the device identifier is 5 bits, a DPA can only map a maximum of 32 PCIe endpoint devices connected to downstream ports. If the number of functions for a PCIe endpoint device is less than 8, the PCIe ID waste problem still exists. To solve this problem, the device identifier of the PCIe endpoint devices mapped by the DPA to its downstream ports can be expanded from 5 bits to 6 bits or 7 bits. Correspondingly, the number of bits occupied by the function is reduced from 3 bits to 2 bits or 1 bit. The table below shows the number of PCIe endpoint devices and PCIe endpoint device functions mapped by the DPA to its downstream ports.

[0186]

[0187] This application provides a PCIe switch including a DPA (Distributed Access Point). The DPA acts as a proxy for multiple downstream ports, presenting them as a single port to the upper-layer system without modifying the existing BIOS, OS PCIe subsystem, or PCIe device drivers. This reduces bus overhead and increases the number of endpoint devices supported by the PCIe system. Specifically, this application can add a DPA driver without changing other parts of the PCIe system software, minimizing impact on the existing system. Furthermore, this application does not require additional circuitry such as an NTB (Network Module) to isolate two PCIe domains, resulting in low cost. Moreover, when the DPA is implemented using an ASIC chip, it offers high performance in forwarding communication between upstream and downstream ports.

[0188] like Figure 12 The diagram shown is a schematic representation of the device structure of a PCIe switching device 1200 according to an embodiment of this application. The PCIe switching device 1200 includes a processor 1201, an internal connection 1202, an upstream port 1204, multiple downstream ports 1205, and a memory 1203. The processor 1201 may have multiple processor cores and may include multiple registers for storing configuration information of the PCIe switching device. Figure 12 The PCIe switching device shown can employ an endpoint agent (DPA) as a software module, meaning that the processor 1201 executes instructions from the memory 1203 to implement the aforementioned DPA function. The upstream port 1202 can be one or more.

[0189] Optionally, the processor 1201 described above may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for controlling the execution of programs according to the present application.

[0190] The aforementioned internal connection 1202 may include a pathway for transmitting information between the aforementioned components. Optionally, the internal connection 1202 is a bus.

[0191] The aforementioned memory 1203 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0192] The memory 1203 stores the application code that executes the scheme of this application, and its execution is controlled by the processor 1201. The processor 1201 executes the application code stored in the memory 1203, and cooperates with the upstream port 1204 and the downstream port 1205, so that the device 1200 can realize the functions described in the foregoing embodiments of this application.

[0193] In a specific implementation, as one embodiment, the processor 1201 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 in the CPU.

[0194] In a specific implementation, as one example, the PCIe switching device 1200 may include multiple processors. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0195] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0196] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A PCIe switching device, characterized in that, include: Upstream port, port proxy device, and multiple first downstream ports, among which, The upstream port is connected to the port proxy device, and the port proxy device is connected to the plurality of first downstream ports, wherein the first downstream ports are used to connect to PCIe endpoint devices; The multiple first downstream ports share the same bus number with their connections to the PCIe endpoint devices below them; Specifically, for different first downstream ports, the BDF of the PCIe endpoint devices under them is different; the number of the first downstream port is used to indicate the device identifier of the PCIe endpoint device connected to the first downstream port.

2. The device as described in claim 1, characterized in that, Also includes: One or more second downstream ports, The one or more second downstream ports are connected to the upstream port.

3. The device as described in claim 1 or 2, characterized in that, The port proxy device and the plurality of first downstream ports have the same home bus number.

4. The device as described in claim 1, characterized in that, The port proxy device is used to receive data packets sent by PCIe endpoint devices from the first downstream port and replace the device identifier in the data packets with the number of the first downstream port.

5. The device as described in claim 4, characterized in that, When the data packet sent by the PCIe endpoint device is a Request for Protocol (TLP) packet, the port proxy device is specifically used to replace the device identifier in the requester's BDF carried in the Request for Protocol (TLP) packet with the number of the first downstream port connected to the PCIe endpoint device, or... When the data packet sent by the PCIe endpoint device is a Complete TLP packet, the port proxy device is specifically used to replace the device identifier in the Completer BDF carried by the Complete TLP packet with the number of the first downstream port connected to the PCIe endpoint device.

6. The device as described in any one of claims 1-5, characterized in that, The port proxy device is used to receive data packets sent by the processor, use the device identifier value in the data packet as the number of the destination port for receiving the data packet, change the device identifier in the data packet to 0, and send the data packet to the destination port.

7. The device as described in claim 6, characterized in that, When the data packet sent by the processor is a configuration request (TLP) packet, the device identifier in the destination BDF carried in the configuration request (TLP) packet is replaced with 0, or... When the data packet sent by the processor is a Complete TLP packet, the device identifier in the requester BDF carried in the Complete TLP packet is replaced with 0.

8. The device as described in any one of claims 1-7, characterized in that, The configuration space register of the port proxy device is mapped with the configuration space registers of all the first downstream ports belonging to the port proxy device.

9. The device as described in any one of claims 1-8, characterized in that, The value of the enable register of the port proxy device is used to indicate whether the proxy function of the port proxy device is enabled.

10. The device as described in any one of claims 1-9, characterized in that, When the PCIe switching device is powered on or reset and initialized, it sets the registers of the port proxy device through firmware or by reading non-volatile memory.

11. The device as described in any one of claims 1-10, characterized in that, The port proxy device is used to receive a first configuration request forwarded by the upstream port, wherein the first configuration request carries the bus numbers corresponding to the management topology of the port proxy device; The port proxy device is further configured to set the bus numbers corresponding to its own management topology according to the first configuration request, and set the management topology of all first downstream ports connected to it to the same bus number.

12. The device as described in any one of claims 1-11, characterized in that, The port proxy device is used to receive a second configuration request sent by the processor. The second configuration request carries a bus number that is the lower-level bus number of the port proxy device and a device identifier that is m, where m is an integer greater than or equal to 0. The port proxy device is further configured to route the second configuration request to the corresponding destination port according to the value m of the device identifier carried in the second configuration request, wherein the number of the destination port is equal to m.

13. The device as described in claim 2, characterized in that, The port proxy device is used to determine, based on the port enable record, that the proxy function is enabled on the plurality of first downstream ports, and to determine that the proxy function is not enabled on the one or more second downstream ports.

14. The device as described in any one of claims 1-13, characterized in that, The address window of the port proxy device is the set of address windows of all first downstream ports connected to it.

15. The device as claimed in claim 1, characterized in that, The port proxy device is also used to receive message packets reported by its first downstream port, replace the BDF of the first downstream port carried in the message packet with its own BDF, and route the message packet to the upstream port.

16. The device as claimed in claim 1, characterized in that, The PCIe switching device is connected to the processor's root port without going through other bridge devices or through other bridge devices.

17. The device as claimed in claim 1, characterized in that, The sum of the number of bits in the device identifier and the number of bits in the function identifier of the PCIe endpoint device is 8.

18. The device as claimed in claim 17, characterized in that, The PCIe switching device is an application-specific integrated circuit (ASIC) chip.

19. The device as described in any one of claims 1-18, characterized in that, The PCIe switching device includes multiple parallel port proxy devices, each of which is connected to a set of first downstream ports.

20. A PCIe switching device, characterized in that, include: The device includes a processor, a memory, an upstream port, and multiple first downstream ports. The memory stores instructions that, when executed by the processor, implement the functions of the port proxy device as described in any one of claims 1-19.

21. A method for extending a PCIe system, characterized in that, The PCIe system includes a processor and a PCIe switching device. The PCIe switching device includes: an upstream port, a port proxy device, and multiple first downstream ports. The port proxy device is connected to the processor via the upstream port and is also connected to the plurality of first downstream ports. The first downstream ports are used to connect PCIe endpoint devices, wherein the BDF (Browser Defender) of the PCIe endpoint devices connected to different first downstream ports is different, and the number of the first downstream port is used to indicate the device identifier of the PCIe endpoint device connected to the first downstream port. The method includes: The processor uses a port proxy device to set the connections of the plurality of first downstream ports and their PCIe endpoint devices to the same bus number.

22. The method as described in claim 21, characterized in that, The port proxy device and the plurality of first downstream ports have the same home bus number.

23. The method as described in claim 21, characterized in that, The method further includes: The port proxy device receives data packets sent by the PCIe endpoint device from the first downstream port and replaces the device identifier in the data packet with the number of the first downstream port.

24. The method as described in claim 21 or 23, characterized in that, The method further includes: The port proxy device receives the data packet sent by the processor, uses the device identifier value in the data packet as the number of the destination port for receiving the data packet, changes the device identifier in the data packet to 0, and sends the data packet to the destination port.

25. The method as described in any one of claims 21-24, characterized in that, The configuration space register of the port proxy device is mapped with the configuration space registers of all the first downstream ports belonging to the port proxy device.

26. The method as described in any one of claims 21-25, characterized in that, The value of the enable register of the port proxy device is used to indicate whether the proxy function of the port proxy device is enabled.

27. The method according to any one of claims 21-26, characterized in that, The method further includes: When the PCIe switching device is powered on or reset and initialized, it sets the registers of the port proxy device through firmware or by reading non-volatile memory.

28. The method according to any one of claims 21-27, characterized in that, The method further includes: The port proxy device receives a first configuration request from the upstream port forwarding, the first configuration request carrying each bus number corresponding to the management topology of the port proxy device; According to the first configuration request, the port proxy device sets the bus number corresponding to its own management topology, and sets the management topology of all first downstream ports connected to it to the same bus number.

29. The method according to any one of claims 21-28, characterized in that, The method further includes: The port proxy device receives a second configuration request sent by the processor. The second configuration request carries a bus number that is the lower-level bus number of the port proxy device and a device identifier that is m, where m is an integer greater than or equal to 0. The port proxy device routes the second configuration request to the corresponding destination port based on the device identifier value m carried in the second configuration request, where the destination port number is equal to m.

30. The method according to any one of claims 21-29, characterized in that, The PCIe switching device further includes one or more second downstream ports, and the method further includes: The port proxy device determines, based on the port enable record, that the proxy function is enabled on the plurality of first downstream ports, and determines that the proxy function is not enabled on the one or more second downstream ports.

31. A PCIe system, characterized in that, Includes a processor and a PCIe switching device as described in any one of claims 1-19.

32. The system as described in claim 31, characterized in that, The PCIe switching device includes a first PCIe switching device and a second PCIe switching device, wherein the upstream port of the second PCIe switching device is connected to the second downstream port of the first PCIe switching device, and the proxy function of the second downstream port is not enabled.

Images