PCIe system extension method and device, cloud server and storage medium

Expanding the PCIe system through multi-stage bridging technology, solving the problem of limiting the number of DPs and buses in the PCIe system, realizing hot-swap management of large-scale elastic equipment and system simplification, and improving the number of equipment and resource utilization.

CN119088740AActive Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310664708.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-12-06
Estimated Expiration
2043-06-06

AI Technical Summary

Technical Problem

The existing PCIe system is limited by the number of DPs and buses in terms of hot plugging, and cannot achieve flexible configuration and management of large-scale elastic devices, resulting in limited number of PF devices and high system management complexity.

Method used

Extend the PCIe system through multi-level bridging technology to generate the target system topology, allowing the second type of DP to mount multiple PF devices, improve the mount capability of a single DP, realize the hot-swap configuration and management of large-scale elastic devices, and reduce bus resources Overhead, improve bus resource utilization.

Benefits of technology

It realizes hot-swap management of large-scale elastic devices, expands the number of hot-swap devices visible to users, simplifies system management, improves bus resource utilization, and reduces system management complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088740A_ABST
    Figure CN119088740A_ABST
Patent Text Reader

Abstract

The invention discloses a method and device for expanding a PCIe system, a cloud server and a storage medium, relates to the technical field of cloud servers, and is used for expanding user-visible hot-plug equipment by adopting a multi-stage bridging technology and realizing hot-plug management of large-scale elastic equipment. According to the embodiment of the invention, the method comprises a first type of DP for mounting a single PF device and a second type of DP for mounting a plurality of PF devices, the second type of DP mounts the plurality of PF devices through a bridge group corresponding to the second type of DP, and the bridge group comprises a PCIe bridge for bridging the second type of DP and a plurality of PCI bridges for mounting the PF devices, that is, through a multi-stage bridging mode, the number of the PF devices capable of being mounted by the single DP is increased, and the number of the PF devices mounted by the single DP is increased. The number of PF devices capable of being provided in the PCIe system is increased, so that hot plug configuration and management of large-scale elastic devices are achieved, and the hot plug devices visible to users are expanded.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to the field of cloud server technology, and provides a method, device, cloud server, and storage medium for extending a PCIe system. Background Art

[0002] Cloud Bare Metal (CBM) is a bare metal cloud service that has both the characteristics of a traditional physical server and the virtualization service functions of cloud computing technology. Bare metal can be simply understood as computer hardware without an operating system, or it can be understood as not completely without an operating system and software, but rather that tenants can choose to configure the required operating system and software. Bare metal servers are physically isolated from other tenants, and therefore have the characteristics of secure physical isolation, which is particularly suitable for scenarios with high security requirements.

[0003] When bare metal cloud services are needed, tenants can choose elastic devices according to their own needs. Accordingly, bare metal cloud service providers can hot-plug or hot-unplug elastic devices according to tenants' choices to meet their needs. Currently, hot-plug technology based on the high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) mechanism is usually used to achieve flexible selection of elastic devices. See Figure 1 As shown, it is a schematic diagram of the hot-swappable topology structure based on the PCIe mechanism, wherein the PCIe switch includes an upstream port (uplink port, UP) and multiple downstream ports (downlink port, DP), and only one hot-swappable device can be mounted under one DP. Usually, a computer system only supports one PCIe domain (domin), and a PCIe domain provides a maximum of 256 bus numbers. The PCIe protocol is a point-to-point transmission protocol, which is limited by the number of DPs and buses, so that the physical function (PF) devices that can be provided in the entire topology are limited.

[0004] Therefore, how to realize hot plugging of large-scale elastic equipment is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, cloud server and storage medium for expanding a PCIe system, which are used to expand user-visible hot-pluggable devices using multi-level bridging technology, thereby realizing hot-plug management of large-scale elastic devices.

[0006] On the one hand, a method for extending a PCIe system is provided, which is applied to a smart network card, and the method includes:

[0007] Obtaining a target number of physical function PF devices required by a target PCIe system to be configured;

[0008] Based on the target number and an upper limit value of the number of pre-configured downstream port DPs, obtain a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system;

[0009] Based on the first number and the second number, a target system topology corresponding to the target PCIe system is generated; wherein the second type DP in the target system topology mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices;

[0010] The target system topology is notified to the host to instruct the host to configure the target PCIe system according to the target system topology.

[0011] In one aspect, a method for extending a PCIe system is provided, which is applied to a host, and the method comprises:

[0012] Receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and multiple PCI bridges mounting PF devices;

[0013] Selecting a PF device that meets the target system topology from the PF devices included in the device, and calling the standard hot-swap controller SHPC of each PF device to initialize the state of each PF device;

[0014] A system identifier is allocated to each PF device respectively, and the allocated system identifier is returned to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

[0015] In one aspect, a smart network card is provided, comprising:

[0016] A receiving unit, configured to obtain a target number of physical function PF devices required by a target PCIe system to be configured;

[0017] A DP configuration unit, configured to obtain, based on the target number and an upper limit value of the number of pre-configured downstream port DPs, a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system;

[0018] A topology generating unit, configured to generate a target system topology corresponding to the target PCIe system based on the first quantity and the second quantity; wherein the second type DP in the target system topology mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices;

[0019] The sending unit is used to notify the host of the target system topology to instruct the host to configure the target PCIe system according to the target system topology.

[0020] In one aspect, a host is provided, comprising:

[0021] A receiving unit, configured to receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices;

[0022] A PCIe system configuration unit, configured to select a PF device that meets the target system topology from the PF devices included in the PCIe system, and to call the standard hot-swap controller SHPC of each PF device to initialize the state of each PF device;

[0023] A resource configuration unit is used to allocate a system identifier to each PF device respectively and return the allocated system identifier to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

[0024] On the one hand, a cloud server is provided, comprising the above-mentioned smart network card and the above-mentioned host.

[0025] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0026] In one aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0027] In one aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the steps of any of the above methods.

[0028] In an embodiment of the present application, a first number of first type DPs and a second number of second type DPs in the target PCIe system are allocated according to the target number of required PF devices to generate a target system topology corresponding to the target PCIe system, and then the host configures the target PCIe system according to the topology. Among them, the second type DP in the target system topology mounts multiple PF devices through its corresponding bridge group, each bridge group includes a PCIe bridge that bridges the second type DP and multiple PCI bridges that mount PF devices, and each PCI bridge can mount multiple PF devices, so a second type DP can realize the mounting of multiple PF devices. It can be seen that through the multi-level bridging method, the number of PF devices that can be mounted by a single DP is expanded, so that the PCIe system is no longer limited to the number of DPs and the number of buses, and the number of PF devices that can be provided in the PCIe system is increased, thereby realizing the hot-swappable configuration and management of large-scale elastic devices, expanding the user-visible hot-swappable devices, and greatly reducing the complexity of system management; and, since a second type DP can mount multiple PF devices, accordingly, the multiple PF devices mounted thereon can share the bus resources of the second type DP, thereby in the scenario of large-scale PF devices, the bus resource overhead can be greatly reduced and the utilization rate of the bus resources can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0030] Figure 1 A schematic diagram of a hot-swap topology structure based on a PCIe mechanism provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0032] Figure 3 An architectural diagram of a bare metal cloud hosting service system provided in an embodiment of the present application;

[0033] Figure 4A flowchart of a method for extending a PCIe system provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a configuration page provided in an embodiment of the present application;

[0035] Figure 6a-6c A schematic diagram of an expandable hot-swap topology based on the SHPC mechanism provided in an embodiment of the present application;

[0036] Figure 7 Another flowchart of the method for extending a PCIe system provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a process for storing device information of a PF device provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the principle of a multi-level table entry query technology based on a CRC encoding method provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of a process for querying device information provided in an embodiment of the present application;

[0040] Fig.11 A flowchart of a hot-swap management process provided in an embodiment of the present application;

[0041] Fig.12 A schematic diagram of the state management principle of a high-reliability hot-swap state machine provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of the structure of a smart network card provided in an embodiment of the present application;

[0043] Fig.14 A schematic diagram of the structure of a host provided in an embodiment of the present application;

[0044] Fig.15 A schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0046] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0047] PCIe system: also called PCIe domain. PCIE is a high-speed serial computer expansion bus standard, PCI is a standard for defining local buses. PCIe bus is a local bus of the processor system, and its function is similar to that of PCI bus, and its main purpose is to connect external devices in the processor system. PCIe bus uses an end-to-end connection method. Only one device can be connected at each end of a PCIe link, and these two devices are data transmitters and receivers for each other.

[0048] A typical PCIe system can be found in Figure 1 As shown, it includes a root complex (RC), a PCIe switch, and a PCIe endpoint device. Among them, the RC can be located on the processor, and the RC itself occupies a bus number. The UP of the PCIe switch is used to connect to the root port (RP) of the RC, and each DP can mount a PF device.

[0049] Smart NIC: Also known as Data Processing Unit (DPU), it is a new type of programmable processor with high processing power and high-performance network interface.

[0050] PF device: also known as flexible device. In the embodiments of the present application, PF device refers to a device with hot-swappable features, such as hard disks, optical drives, printers, network cards, graphics cards, sound cards, expansion cards, etc. These devices can be added or removed during system operation without affecting the normal operation of the system.

[0051] Bridge (BR): refers to a bridge device in a PCIe or PCI topology. The embodiments of the present application mainly involve a PCIe-PCI bridge (referred to as PCIe bridge) and a PCI-PCI bridge (referred to as PCI bridge). The PCIe bridge implements the communication between the PCIe bus protocol device and the PCI bus protocol device, and is used to distribute control messages to the lower-level PCI bridge. The PCI bridge implements the communication between the PCI bus protocol device and the PCI bus protocol device, and is used to distribute control messages to the lower-level PCI bridge or PF device. Thus, through the combination of the PCIe bridge and the PCI bridge, multiple PF devices under the same DP are visible to users, and management of multiple PF devices is achieved.

[0052] Cyclic Redundancy Check (CRC) algorithm: It is a fast algorithm that generates a short fixed-bit check code based on network data packets or computer files. It is a commonly used check code in the field of computer network communications. CRC uses the principle of division and remainder. The CRC code includes a series of data encoding rules such as shift and division. Its algorithm principle, algorithm program design and analysis can all be solved through corresponding software coding. The cyclic redundancy check code is an algorithm that uses software for verification, so its verification speed is very fast, the bit error rate of the verification is also low, and the information transmission speed of the entire computer network communication is very high. It has the advantages of clear principle and simple implementation.

[0053] Standard Hot Plug Controller (SHPC): It is used to implement the management and control of hot-plug devices. The Peripheral Component Interconnect Special Interest Group (PCI SIG) has developed the PCI hot-plug specification, which defines the platform, board, and software elements necessary to support hot-plug. Later, PCI SIG launched the SHPC specification, which clarifies the standard usage mode and strict register group requirements for hot-plug, and allows operating system providers to provide hot-plug support in addition to platform-specific software, gradually completing the hot-plug standard formulation work.

[0054] System ID: refers to the unique ID of each device or module contained in the PCIe system. Taking the PF device as an example, the system ID can be the unique ID of the PF device in the PCIe system. Generally speaking, the system ID can be at least one of the memory address ID, base address register (BAR) address or bus number, device number and function number combination (Bus-Device-Function, BDF) information assigned to the PF device.

[0055] Bare metal cloud servers are non-virtualized servers that run directly on hardware without an intermediate virtualization layer. Compared with traditional virtualized servers, bare metal servers can provide higher performance and lower latency, thereby providing enterprises with exclusive physical servers on the cloud, providing excellent computing performance and data security for core databases, key application systems, high-performance computing, big data and other businesses, allowing cloud service tenants to apply flexibly and use on demand.

[0056] When bare metal cloud services are needed, tenants can choose elastic devices according to their own needs. However, usually, a computer system only supports one PCIe domain and provides 256 bus numbers for each PCIe domain, that is, the entire computer system only provides 256 bus numbers. Since the PCIe protocol is a point-to-point connection, the scale of PCIe devices that can be supported by a PCIe domain is restricted by the number of 256 bus numbers. It is impossible to support more PCIe devices in a PCIe domain. In actual applications, other PCIe devices except PF devices will also occupy bus numbers. Therefore, the number of PF devices is always less than 256, making the number of PF devices available in the entire topology extremely limited.

[0057] In order to make up for the problem of insufficient PF number, the relevant technology proposes that the single root I / O virtualization (SR-IOV) technology can be used to virtualize the virtual function (VF) device to carry the elastic device. However, the VF device generated by the SR-IOV technology needs to set the feature bits and load the driver after enabling SR-IOV, and the reverse operation is required when uninstalling. It is not user-friendly and is easy to cause the failure of adding or deleting devices due to the wrong sequence. At the same time, modifying the feature bits will also invade the host image. Therefore, it is difficult to implement in the public cloud bare metal cloud host products.

[0058] Therefore, how to realize hot plugging of large-scale elastic equipment is still a problem that needs to be solved urgently.

[0059] Based on this, an embodiment of the present application provides a method for expanding a PCIe system, in which a first number of first type DPs and a second number of second type DPs in a target PCIe system are allocated according to a target number of required PF devices to generate a target system topology corresponding to the target PCIe system, and then the host configures the target PCIe system according to the topology. Among them, the second type DP in the target system topology mounts multiple PF devices through its corresponding bridge group, each bridge group includes a PCIe bridge that bridges the second type DP and multiple PCI bridges that mount PF devices, and each PCI bridge can mount multiple PF devices, so a second type DP can realize the mounting of multiple PF devices. It can be seen that through the multi-level bridging method, the number of PF devices that can be mounted by a single DP is expanded, so that the PCIe system is no longer limited to the number of DPs and the number of buses, and the number of PF devices that can be provided in the PCIe system is increased, thereby realizing the hot-plug configuration and management of large-scale elastic devices, expanding the user-visible hot-pluggable devices, and there is no need to generate VF devices through SR-IOV technology, so that the complexity of system management is greatly reduced; and since a second type DP can mount multiple PF devices, accordingly, the multiple PF devices mounted thereon can share the bus resources of the second type DP, thereby in the scenario of large-scale PF devices, the bus resource overhead can be greatly reduced and the utilization rate of bus resources can be improved.

[0060] The above method greatly expands the number of PF devices, while also bringing about efficiency issues in PF device resource management. Therefore, in order to solve the problem of fast indexing of large-scale PF devices, an embodiment of the present application provides a multi-level table item query method, in which each table item index consists of one or more index items. If all entries corresponding to the table item index are unavailable, the next level table item index is switched to for reading and writing, thereby meeting the needs of indexing a large number of PF devices.

[0061] In addition, a high-reliability hot-swap state machine is designed in the embodiment of the present application to ensure that the final state of the PF device is consistent with the user's expected state and to ensure high availability. Specifically, the high-reliability hot-swap state machine takes into account multiple scenarios such as the host side not being turned on, the host restarting, the hot-swap interruption being lost, and the user repeatedly hot-plugging. The state machine rotates according to the user's expected state and the current PF device state to ensure that the device state finally presented to the user is the state expected by the user, thereby improving the user experience.

[0062] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0063] The solution provided in the embodiment of the present application can be applied to bare metal server scenarios. Figure 2 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a terminal device 10 and a cloud server 20 may be included.

[0064] The terminal device 10 can be any terminal device such as a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart vehicle-mounted device, and a smart wearable device. The terminal device 10 can be installed with a target application, and the target should have the function of configuring and presenting a bare metal cloud host. The application involved in the embodiment of the present application can be a software client, or a client such as a web page or a small program, and the specific type of the client is not limited.

[0065] The server 20 is used to deploy and manage the bare metal cloud host and provide bare metal cloud host related services to users. The server 20 can be, for example, an independent computer device or a physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited thereto.

[0066] See also Figure 2 As shown, the server 20 provides a bare metal cloud host service, and may include a smart network card 201 and a host 202 that are communicatively connected to each other. For example, the smart network card 201 may be inserted into a corresponding card slot in the host 202. The smart network card 201 has an independent computing unit. The above-mentioned PCIe system is actually deployed and implemented by the host 202. The host 202 includes a series of selectable PF devices, and the PCIe system may be deployed according to the user's selection.

[0067] It should be noted that the method for extending the PCIe system in the embodiment of the present application can be implemented by combining the smart network card 201 and the host 202. Specifically, the smart network card 201 can decide which topology to use for deployment based on the user's requirements for the PCIe system, such as indicating the number of PF devices required. When the number of PF devices exceeds the upper limit of the DP, it is impossible to use the traditional topology solution. Therefore, some of the DPs can be selected to mount multiple PF devices, and a system topology containing this type of DP is generated for the user, and notified to the host 202 for corresponding deployment. In actual scenarios, the number of smart network cards 201 and hosts 202 included in the server 20 can be one or more, and the embodiment of the present application does not limit this.

[0068] In the embodiment of the present application, the terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more networks. The network may be a wired network or a wireless network, for example, a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, or other possible networks, which are not limited in the embodiment of the present application.

[0069] like Figure 3 As shown, it is an architecture diagram of a bare metal cloud host service system provided by an embodiment of the present application. In this architecture, it can include a client part and a server part. The client part is used to provide users with pages for configuring and using cloud host services, and the server part is composed of a smart network card and a host. The smart network card is used to implement the management of cloud host services, and the host is a device that provides actual cloud host services to users. For example, when a user needs to configure multiple hot-swappable network cards, the host can be a device containing multiple network cards, and the required number of network cards can be configured for the user based on the user's choice to implement the corresponding network service.

[0070] See also Figure 3 As shown, the smart network card includes a topology management module, a table query module and a hot-swap status management module. Among them, the topology management module is used to generate a system topology of the corresponding PCIe system for the user based on the number of PF devices configured by the user, and notify the host to configure according to the system topology, and present the system topology to the user, so as to present large-scale PF devices to the user. The table query module provides a multi-level table query method based on large-scale PF devices, which is used to quickly query the relevant information of the required PF device when it is necessary to query the device information of the PF device, such as routing information. The hot-swap status management module is used to manage the hot-swap status of the PF device to ensure that the status of each PF device is highly consistent with the user's expected status.

[0071] The host includes a CPU, SHPC, and PF devices. The PF device is flexibly configured based on the user's selection, and the SHPC is used to set the status of the PF device.

[0072] In actual application, the user can configure the system configuration information of the target PCIe system that he wants to configure in the configuration page provided by the client, such as the target number of PF devices that the system needs to include, and then the smart network card can generate the target system topology corresponding to the target PCIe system based on the target number and the method for extending the PCIe system provided in the embodiment of the present application, and notify the host to perform corresponding configuration according to the target system topology, so as to configure the target PCIe system required by the user and provide relevant services to the user. For example, when the number of PF devices required by the user is huge, the method for extending the PCIe system provided in the embodiment of the present application can be adopted, and a multi-level bridging method can be adopted to select some DP bridges to mount multiple PF devices. The PF devices mounted on the same DP share the bus number corresponding to the DP, and a bridge group is connected to each DP to facilitate the management of these PF devices.

[0073] Taking into account the problem of low efficiency in indexing large-scale PF devices, the table item query module of the embodiment of the present application encodes the system identifier of the PF device as the table item index of the PF device, and when the table item index is full, jumps to the next level table item index to continue storing the table items. Correspondingly, when performing a table item query, the system identifier of the PF device to be queried is encoded in the same way, and the table is queried level by level according to the obtained table item index until the relevant information of the PF device is found. By means of index query, the scope of the table query can be narrowed as much as possible, thereby improving the index efficiency of the PF device.

[0074] The following describes the method of extending the PCIe system provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this respect.

[0075] See also Figure 4 As shown, it is a flow chart of a method for extending a PCIe system provided in an embodiment of the present application. The method can be executed by a smart network card. The specific implementation process is as follows:

[0076] Step 401: Obtain the target number of physical function PF devices required by the target PCIe system to be configured.

[0077] In the embodiment of the present application, the target quantity may be input by the target object in the configuration page of the PCIe system, for example, when configuring a new PCIe system, or when modifying an already created PCIe system. Figure 5 As shown, it is a schematic diagram of the configuration page provided in an embodiment of the present application, which includes configuration items for PF type selection and PF quantity. The target object can select the type of PF device and the number of PF devices required for the corresponding type according to its own needs, and then initiate a configuration request for the target PCIe system, so that the smart network card can obtain the target number of physical function PF devices required for the target PCIe system to be configured based on the configuration request.

[0078] It should be noted that the smart network card can directly receive the configuration request sent by the target object, or it can receive the configuration request through other devices, for example, it can receive the configuration request forwarded by the access device of the cloud server. In addition, the target object refers to a user in the cloud host service platform, which can refer to a natural person or an account representing a user identity, such as represented by an account identifier. The embodiment of the present application does not impose too many restrictions on the target object.

[0079] Step 402: Based on the target number and the upper limit value of the number of pre-configured DPs, obtain a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system.

[0080] The upper limit of the number of pre-configured downstream ports DP may be, for example, the maximum number of DPs that a PCIe system can contain. For example, a PCIe system may contain 32 DPs. According to the solution in the related art, only one PF device can be mounted under one DP. Therefore, due to the number of DPs and buses, the number of PF devices that a PCIe system can provide is limited, which obviously greatly limits the number of PFs that can be presented in the system.

[0081] Therefore, the embodiment of the present application provides a scalable hot-swap technology based on the SHPC mechanism. By adopting a multi-level bridging approach, multiple PF devices can be mounted for each DP to achieve large-capacity, high-density, and highly reliable elastic devices, and directly present PF devices to users.

[0082] See also Figure 6aAs shown, it is a schematic diagram of an expandable hot-swappable topology based on the SHPC mechanism provided by an embodiment of the present application, wherein, to support SHPC hot-swappable, the embodiment of the present application needs to bridge the PCIe bridge under the DP, and bridge the PCI bridge under the PCIe bridge, that is, the second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge that bridges the second type DP and multiple PCI bridges that mount PF devices. In this way, for each PCI bridge, multiple hot-swappable PF devices can be mounted under one PCI bridge, and usually up to 32 PF devices can be mounted. Then, the target object can select multiple PCI bridges as needed, or after the target object enters the target number of required PF devices, the smart network card will match the target object with an appropriate number of PCI bridges to achieve flexible expansion of elastic devices.

[0083] See also Figure 6a As shown in, DP includes two types, one is the first type DP for mounting a single PF device, and the other is the second type DP for mounting multiple PF devices through multi-level bridging technology. In actual scenarios, the first number of the first type DP and the second number of the second type DP can be flexibly configured according to the target number to meet the target number of PF devices required by the target object.

[0084] In one possible implementation, in order to ensure the reliability and stability of the PCIe system, the number of first type DPs can be increased as much as possible, that is, the first number of first type DPs and the second number of second type DPs can be determined by maximizing the first type DP and combining the target number and the above-mentioned upper limit value.

[0085] Exemplarily, when the upper limit of the number of DPs is O, then if permitted, one of the DPs can be adopted as the second type DP, and the remaining DPs are all first type DPs, that is, the number of first type DPs is (O-1) and the number of second type DPs is 1.

[0086] In a possible implementation, in order to improve the reliability and stability of the PCIe system, constraints can also be imposed on the number of PF devices that can be mounted on the second type DP, that is, to avoid too many PF devices mounted on the same DP. When allocating the first type DP and the second type DP, the target number and the upper limit of the number of DPs and the upper limit of the number of PF devices mounted on each DP can be combined to obtain the first number of the first type DP and the second number of the second type DP.

[0087] Exemplarily, when the total number of PF devices that need to be mounted is Q, the upper limit of the number of DPs is O, and each DP can mount a maximum of P PF devices, the first number and the second number can be determined by traversal. That is, first, it can be traversed to see whether the remaining (O-1) DPs can meet the mounting requirements of (QP) PF devices when 1 second-type DP is used. If so, the number of first-type DPs is (O-1) and the number of second-type DPs is 1; if not, the number of second-type DPs is increased by one, and similarly, it is determined whether the remaining (O-2) DPs can meet the mounting requirements of (Q-2P) PF devices. If so, the number of first-type DPs is (O-2) and the number of second-type DPs is 2; if not, the number of second-type DPs continues to increase by one, and the above process is repeated until the mounting requirements are met.

[0088] In the embodiment of the present application, the second type DP mounts multiple PF devices through a multi-level bridging technology, which means that each second type DP mounts a bridge group, and multiple PCI bridges in each bridge group adopt a tree topology structure, see Figure 6b As shown, it is a schematic diagram of the tree topology structure provided by an embodiment of the present application, and the tree topology structure includes N levels of PCI bridges, and each level of PCI bridge includes at least one PCI bridge, wherein, as shown in Figure b, the first level PCI bridge is used to bridge the PCIe bridge and its corresponding child node in the second level PCI bridge, that is, it is used to connect the PCIe bridge and the PCI bridge under the PCI bridge; the i-th level PCI bridge is used to bridge its corresponding parent node in the i-1-th level PCI bridge and its corresponding child node in the i+1-th level PCI bridge, where i is an integer greater than 1 and less than N; the N-th level PCI bridge bridges its corresponding parent node in the N-1-th level PCI bridge and at least one PF device mounted thereon, thereby realizing a second type DP mounting multiple PF devices through multi-level bridging.

[0089] In an embodiment of the present application, the number N of levels of the tree topology structure can be determined based on the total number of PF devices that need to be mounted on the corresponding second type DP, so that the number and levels of PCI bridges can be flexibly configured according to the total number of PF devices that need to be mounted to meet the target object's demand for PF devices and realize large-scale deployment and presentation of PF devices. When the PCIe system is presented to the target object through the above-mentioned tree topology structure, the target object can clearly perceive the architecture of the entire PCIe system, thereby facilitating hot-plug operations on the PF device.

[0090] In a possible implementation, when configuring the first quantity and the second quantity, the level N of the tree topology structure can also be considered at the same time. The level N represents the depth of the tree topology structure to a certain extent. The deepening of the depth will bring certain difficulties to the subsequent management and indexing. Once you can increase the upper limit of the level N, then when determining the first quantity and the second quantity, you need to consider the upper limit value of the larger level N, which is similar to the process of considering the upper limit value of DP mentioned above, so it will not be repeated here.

[0091] Of course, the target object may also choose the ratio or preference between the first type DP and the second type DP, and the first quantity and the second quantity may also be determined according to the choice of the target object.

[0092] See also Figure 6a As shown, it shows a hot-swap system topology when the level number N is 1, that is, in this system topology, only one level of PCI bridge is included, and these PCI bridges are all connected to the PCIe bridge and connected to the PF device.

[0093] See also Figure 6c As shown, another schematic diagram of an expandable hot-swap topology based on the SHPC mechanism is shown, which is a hot-swap system topology when the level N is 2, that is, in this system topology, for a second type DP, its corresponding bridge group includes two levels of PCI bridges, wherein the first-level PCI bridges are all connected to the PCIe bridge, and are connected to multiple PCI bridges in the second-level PCI bridge, and the second-level PCI bridge is connected to multiple PF devices. By expanding the number of levels of PCI bridges, the number of PF devices that can be mounted can be greatly increased, and the number of PF devices displayed to the target object can be increased.

[0094] Step 403: Based on the first quantity and the second quantity, generate a target system topology corresponding to the target PCIe system; wherein, in the target system topology, the second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge that bridges the second type DP and multiple PCI bridges that mount PF devices.

[0095] In the embodiment of the present application, after determining the first quantity and the second quantity, a target system topology corresponding to the target PCIe system can be generated based on the first quantity and the second quantity. In the target system topology, a first quantity of first-type DPs is included, and each of these first-type DPs is mounted with a PF device, and a second quantity of second-type DPs is included, and each of these second-type DPs is mounted with a bridge group, and each bridge group is as follows: Figure 6b As shown, it includes a PCIe bridge for bridging the second type DP and a plurality of PCI bridges for mounting PF devices.

[0096] Step 404: notify the host of the target system topology to instruct the host to configure the target PCIe system according to the target system topology.

[0097] Specifically, the smart network card generates corresponding system configuration information based on the target system topology generated above, and sends it to the host side, so that the host side can configure a target PCIe system expected by the target object according to the target system topology. It should be noted that the system configuration information may include other configuration information in addition to the target system topology, and the embodiments of the present application do not limit this.

[0098] Correspondingly, for the host side, after receiving the target system topology, it can perform corresponding configuration. Therefore, the following describes the method process for the host side, see Figure 7 As shown, it is another flow chart of a method for extending a PCIe system provided by an embodiment of the present application. The method can be executed by a host, and the specific implementation process is as follows:

[0099] Step 701: Receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge that bridges the second type DP and multiple PCI bridges that mount PF devices.

[0100] See also Figure 6a to Figure 6c As shown, the system topology provided in the embodiment of the present application includes a first type DP for mounting a single PF device and a second type DP for mounting multiple PF devices. It can be flexibly configured in actual scenarios. For each second type DP, it can mount multiple PF devices through its corresponding bridge group. The bridge group includes a PCIe bridge for bridging the second type DP and multiple PCI bridges for mounting PF devices. For details, please refer to the introduction of the relevant parts above, which will not be repeated here.

[0101] Step 702: Select a PF device that meets the target system topology from the PF devices included in the system, and call the standard hot-swap controller SHPC of each PF device to initialize the status of each PF device.

[0102] In the embodiment of the present application, the host side includes enough PF devices for users to choose from. After the user selects a certain number of PF devices, the PCIe system will be configured according to the number selected to meet the user's expectations.

[0103] Specifically, based on the received target system topology, the host side can determine which of its remaining PF devices can still meet the requirements of building the target system topology, thereby allocating these PF devices to the target object and calling the corresponding SHPC to initialize the status of each PF device, such as initializing the status of these PF devices to the inserted state.

[0104] Step 703: Allocate a system identifier to each PF device respectively, and return the allocated system identifier to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

[0105] The system ID is used to distinguish different PF devices, and the subsequent search for PF devices can be performed based on the system ID.

[0106] Generally speaking, in a PCIe system, a 32-bit PCIe ID can be used to identify a PCIe device in the system, namely, a PF device. The 32 bits include the following:

[0107] (1) Domain information, i.e., PCIe domain identifier, occupies bits.

[0108] (2) Bus ID, occupies 8 bits.

[0109] (3) Device ID, occupies 5 bits.

[0110] (4) Function ID, occupies 3 bits.

[0111] Among them, bus id, device id and function id are collectively called BDF information.

[0112] In the embodiment of the present application, the process of allocating system identification to each PF device can be performed during initialization, or can be allocated or updated during subsequent use, such as when the host side is restarted or reset. Of course, the system identification can also include other information, which is not limited in the embodiment of the present application.

[0113] In the embodiment of the present application, in order to facilitate the data plane interaction between PF devices, or between PF devices and other devices, the device indexing can be quickly performed, and the embodiment of the present application also provides a method for quickly indexing PF devices, see Figure 8 As shown, it is a schematic diagram of the process of storing device information of PF devices provided in an embodiment of the present application.

[0114] Step 801: In response to a table entry update instruction, obtain a system identifier of a target PF device to be stored; wherein the system identifier is generated based on a target system topology and is a unique identifier of the target PF device in a target PCIe system.

[0115] In the embodiment of the present application, in order to facilitate the rapid acquisition of device information of each PF device during subsequent data processing, the smart network card will store and maintain the device information of each PF device, and when the device information needs to be used, it will be quickly indexed by the system identifier of the PF device. The device information may be, for example, the routing information of the PF device, the sending queue number, etc.

[0116] Specifically, when the target PF device is initialized or information changes, the table entry corresponding to the target PF device may be updated, and the smart network card may receive the table entry update instruction, and further obtain the system identifier of the target PF device.

[0117] In a possible implementation, when the host side allocates new device resource information to the PF device, it can notify the smart network card to update the table entry, so that the host side can send a table entry update instruction to the smart network card, and carry the system identifier of the target PF device. For example, the system identifier can be output to the smart network card through the Basic Input Output System (BIOS) of the host side, such as the memory address, BAR address and BDF information of the target PF device.

[0118] In a possible implementation, the table entry update instruction may also be triggered by the target object through the client, and the smart network card may also obtain the system identifier of the target PF device from the BIOS on the host side based on the table entry update instruction.

[0119] Step 802: Use a preset data encoding method to encode the system identifier to obtain an initial table entry index corresponding to the target PF device.

[0120] In the embodiment of the present application, the data encoding method may adopt any possible data compression encoding method, such as CRC encoding method, Hash encoding method, etc. Taking the CRC encoding method as an example, the smart network card may calculate the CRC index obtained according to the system identifier of the received PF device, that is, the table entry index corresponding to the target PF device.

[0121] Step 803: Determine whether the number of PF devices associated with the initial entry index has reached the upper limit, that is, determine whether the initial entry index is available.

[0122] Generally speaking, the number of entries that can be associated and stored with a table entry index is limited. The upper limit of this number is called the depth of a table entry index. Therefore, after generating the initial table entry index of the PF device, it is necessary to determine whether the initial table entry index has reached the corresponding depth to determine the availability of the initial table entry index.

[0123] See also Fig. 9 As shown, it is a schematic diagram of the principle of a multi-level table entry query technology based on a CRC encoding method provided by an embodiment of the present application. Among them, each table entry index index is composed of one or more table entries. When all entries corresponding to the index are unavailable, the next level table entry index is switched to read and write. Fig. 9 As shown, each index corresponds to n entries, where n is the depth of each index. If each entry can be used to store device information of a PF device, each index can correspond to n associated PF devices.

[0124] Then, when the information of a PF device needs to be stored, the system identifier of the PF device is encoded by the CRC encoding method to obtain the initial table entry index, that is, Fig. 9 As shown in the index1, it can be determined whether the n entries corresponding to index1 are available, that is, whether they have been occupied by other PF devices. If they are available, they can be used to store the device information of the current PF device. Otherwise, index1 cannot be used for storage, and a shift process is performed to obtain the next level index2, and continue to determine whether index2 is available until an available index is found.

[0125] Step 804: If the result of step 803 is no, the target PF device is associated with the initial table entry index, and the table entry corresponding to the target table entry index is updated based on the device information of the target PF device.

[0126] Specifically, when the upper limit value is not reached, it indicates that the initial table entry index is available, and all information corresponding to the index can be updated. For example, if the device information to be stored is the routing information of the target PF device, such as the bar address, pfindex, and valid mark, etc., then if the initial table entry index is available, this information is stored in the table entry corresponding to the initial table entry index. Of course, the device information may also include any other possible information, and the embodiments of the present application do not limit this.

[0127] Step 805: If the result of step 803 is yes, then obtain the target entry index whose number of associated PF devices does not reach the upper limit from the entry index cascaded from the initial entry index.

[0128] Specifically, the table entry indexes cascaded from the initial table entry index are traversed until a target table entry index whose number of associated PF devices does not reach the upper limit is obtained; since each traversal process is similar, this traversal is specifically introduced here as an example.

[0129] During each traversal, the current table entry index is shifted to obtain the next-level table entry index corresponding to the current table entry index, and it is determined whether the number of associated PF devices of the next-level table entry index has reached the upper limit. If it has not reached the upper limit, the next-level table entry index is the target table entry index. If the number of associated PF devices of the next-level table entry index has reached the upper limit, the next traversal process is entered, and the above process is repeated until the target table entry index that has not reached the upper limit is found.

[0130] Among them, when it is the first traversal, the current table entry index is the initial table entry index, and when it is not the first traversal, the current table entry index is the table entry index obtained by shift processing during the previous traversal.

[0131] Step 806: associate the target PF device with the target table entry index, and update the table entry corresponding to the target table entry index based on the device information of the target PF device.

[0132] It should be noted that the table entry query module in the smart network card includes a software part, and the above process of storing table entries can be implemented by the software part.

[0133] In actual scenarios, if the system identification or device information of the PF device changes, such as when it is updated during hot plugging, the originally stored content needs to be deleted.

[0134] In the embodiment of the present application, when the device information of the target device is stored, the device information can be obtained by querying when it is needed later. Therefore, the embodiment of the present application provides a method for multi-level table item query, see Fig.10 As shown, it is a schematic diagram of the process of device information query provided in the embodiment of the present application. The method can also be implemented by the smart network card in the embodiment of the present application.

[0135] Step 1001: In response to a device query instruction, obtain the system identification of the target PF device to be queried.

[0136] In the embodiment of the present application, when the device information stored in the smart network card needs to be used, the smart network card can be requested to query the device information. For example, when the PF device is a network card, if network card 1 needs to send a data message to network card 2, it is necessary to query the routing information of network card 2 through the smart network card to distribute the data message to the corresponding message queue, or when other devices need to send messages to the PF device, it is also necessary to query the routing information of the PF device through the smart network card to distribute the message to the corresponding message queue.

[0137] Step 1002: Use a data encoding method to encode the system identifier and obtain a query table entry index corresponding to the target PF device.

[0138] Similar to the storage process, when performing a table item query, the system identifier of the input target PF device is also encoded based on the same data encoding method as in the storage process to obtain the query table item index used for the query process.

[0139] Step 1003: Match the target PF device with the associated PF device indexed by the query entry to determine whether the match is successful.

[0140] Step 1004: If the match is successful, the device information of the target PF device is obtained based on the query table item index, that is, the device information of the target PF device is stored in the table item corresponding to the query table item index, so the corresponding device information can be obtained from the table item corresponding to the query table item index.

[0141] Step 1005: If the match is not successful, a shift process is performed to obtain the next level entry index.

[0142] Step 1006: Match the target PF device with the associated PF device indexed by the next level table entry to determine whether the match is successful.

[0143] Step 1007: If the match is successful, then the device information of the target PF device is obtained based on the index of the successfully matched target entry. If the match is not successful, then jump to step 1005 to continue execution.

[0144] Specifically, the table item query process still adopts a multi-level table item query method, that is, the target PF device is matched with the associated PF devices of the query table item index or its cascaded table item index in turn until a successfully matched target table item index is obtained, and the device information of the target PF device is obtained based on the target table item index.

[0145] Taking the CRC encoding method as an example, first, CRC calculation is performed based on its system identifier to obtain the corresponding CRC code, and it is used as the index to determine whether the PF device can be successfully matched in the table item corresponding to the index. If it cannot be matched successfully, shift processing is performed based on the CRC code to obtain a new index, and match again, and so on, until the device information of the PF device is obtained.

[0146] In an embodiment of the present application, the elastic device hot-swap process implemented by an extensible hot-swap device based on the SHPC mechanism involves multiple components, the process is long, and the scenarios are complex and diverse. In order to simplify the user application program interface (Application Program Interface, API) and improve the user experience, in order to ensure that the hot-swap status of each PF device reaches the user's expected state, an embodiment of the present application also provides a high-reliability hot-swap state machine. The state machine takes into account multiple scenarios such as the host side not being turned on, the host restarting, the hot-swap interruption being lost, and the user repeatedly hot-plugging. The state machine is rotated according to the user's expected state and the current state of the PF device to ensure that the hot-swap state finally presented to the user is the user's expected state, thereby improving the user experience.

[0147] See also Fig.11 As shown, it is a flow chart of the hot-swap management process provided in an embodiment of the present application. The method can be implemented by combining the smart network card and the host side.

[0148] Step 1101: the smart network card sends the target system topology to the target object corresponding to the target PCIe system, so as to present the target system topology in the terminal device corresponding to the target object.

[0149] That is, after the target system topology is generated, or when the target object queries the topology information, the target system topology can be presented to the target object. By using the system topology provided in the embodiment of the present application, the PCIe system containing a large number of PF devices can be directly presented to the target object, and the topology of the PCIe system is also clearer, which is convenient for management.

[0150] Step 1102: the smart network card receives a hot-plug control request sent by the target object, where the hot-plug control request is used to request setting a target PF device in the target system topology to a target hot-plug state.

[0151] In actual application, the target object can operate on the PF device for which it has operation authority to switch the hot plug state of the PF device, for example, hot plugging the PF device to set it to the inserted state, or hot unplugging the PF device to set it to the unplugged state.

[0152] Step 1103: the smart network card sends a hot-plug control message carrying the system identifier of the target PF device to the host. The hot-plug control message carries the system identifier of the target PF device and indicates to set the target PF device to a target hot-plug state.

[0153] Step 1104: The host side determines the target SHPC corresponding to the target PF device based on the target system topology and the system identifier, and calls the target SHPC to configure the target PF device to the target hot-swap state.

[0154] In the embodiment of the present application, the host side can search step by step in the target system topology based on the system identifier until the corresponding target PF device is found, and call its corresponding target SHPC to configure the target PF device to the target hot-plug state. For example, the host side finds the corresponding PCIe bridge according to the system identifier, and then the PCIe bridge finds the PCI bridge corresponding to the current PF device from the PCI bridges it is connected to, and finally finds the current PF device.

[0155] Specifically, the high-reliability hot-swap state machine in the smart network card stores the hot-swap status of each PF device. When the smart network card receives a hot-swap control request, it can combine the hot-swap status of the target PF device stored in itself and the status of the host side to determine the hot-swap control method for the target PF device.

[0156] See also Fig.12 As shown, the state management principle diagram of the high-reliability hot-swap state machine provided by the embodiment of the present application. Among them, the hot-swap state of the PF device mainly includes the unplugged state (Slot disable), the plugged state (Slot enabled) and two intermediate states, namely the hot-adding state (hot adding) and the hot-removing state (hot removing). When the PF device is in different states, different processing methods need to be adopted to initiate the hot-swap operation, which are explained separately below.

[0157] (1) The target hot-swap state expected by the target object is the Slot enabled state, that is, the operation initiated by the target object on the target PF device is a hot-plug operation:

[0158] In the first case, see Fig.12As shown, that is, based on the status information stored in itself, when it is determined that the hot-swap status of the target PF device is the Slot disable status, it is necessary to detect the running status of the host. If the host is in the power-on state, the smart network card sends a hot-plug control message to the host side, such as sending an MSI (message signal interrupt) interrupt signal to the host side to indicate that a hot-plug operation is to be performed on the target PF device. At the same time, the smart network card updates the hot-plug status of the target PF device in the status information stored in itself to hot adding, and the host side starts to load the driver of the PF device until it is detected that the driver of the target PF device is loaded successfully. In response to the indication that the driver of the target PF device is loaded successfully, the hot-plug status of the target PF device is updated to Slot enabled, which is visible to the host side.

[0159] In the second case, see Fig.12 As shown, based on the status information stored in itself, it is determined that the hot-plug status of the target PF device is not inserted, and the host is in a non-powered (host off) state. For example, when the host side is not powered on or the host is restarted, the hot-plug status of the target PF device is directly updated to the target hot-plug status, and the host side is waited for to be powered on. When it is detected that the host is powered on, a status recovery message is sent to the host side to instruct the hot-plug status of the target PF device to be restored to the target hot-plug status.

[0160] For the third case, see Fig.12 As shown, when it is determined that the hot-plug state of the target PF device is the Slot enabled state or the hot adding state based on the state information stored in itself, the current state is still maintained and the hot-plug operation performed by the target object is not responded.

[0161] (2) The target hot-swap state expected by the target object is the Slot disable state, that is, the operation initiated by the target object on the target PF device is a hot-unplug operation:

[0162] In the first case, see Fig.12 As shown, that is, based on the state information stored in itself, when it is determined that the hot plug state of the target PF device is the Slot disable state or the hot removing state, the current state is still maintained and the hot plug operation performed by the target object is not responded.

[0163] In the second case, see Fig.12As shown, that is, based on the status information stored in itself, when it is determined that the hot-plug status of the target PF device is the Slot enabled status, it is necessary to detect the running status of the host. If the host is in the power-on state, a hot-plug control message is sent to the host, such as sending an MSI interrupt signal to the host side to instruct the hot-plug operation to be performed on the target PF device. At the same time, the smart network card updates the hot-plug status of the target PF device to the hot removing status, and the host side starts to uninstall the driver of the PF device until it is detected that the driver of the target PF device is successfully uninstalled. Then, in response to the indication of successful driver uninstallation of the target PF device, the hot-plug status of the target PF device is updated to the Slot disable status, that is, the target PF device is unplugged and is invisible to the host side.

[0164] For the third case, see Fig.12 As shown, based on the status information stored in itself, when it is determined that the hot-swap status of the target PF device is Slot enabled and the host side is not powered on, the hot-swap status of the target PF device is directly updated to the target hot-swap status, and the host side is waiting for startup. When it is detected that the host is powered on, a status recovery message is sent to the host side to instruct the hot-swap status of the target PF device to be restored to the target hot-swap status.

[0165] In addition, the smart network card will periodically detect the actual status of the PF device on the host side. When an abnormality occurs, such as the loss of the MSI interrupt signal, which makes the actual status of the host side inconsistent with the status stored in itself, the status of the host side will be adjusted. At the same time, if the user receives hot-plug operations multiple times in a short period of time, it will respond selectively, such as responding to the first or last operation.

[0166] To sum up, in the embodiments of the present application, a scalable hot-swappable technology based on the SHPC mechanism is provided, which realizes large-capacity, high-density, and highly reliable elastic devices, and directly presents large-scale elastic hot-swappable devices to users. Compared with the traditional hot-swappable mechanism, the bus resource overhead is greatly reduced, and the bus resources can be reduced to 1 / 30 of the original. A large number of PF devices can directly meet the user's elastic device usage needs without the need to rely on the VF provided by the SRIOV technology, and the user experience and system management complexity are greatly reduced; at the same time, the multi-level table query technology provided in the embodiments of the present application improves the query efficiency in large-scale PF device scenarios, and the addition of the state machine simplifies user operations and internally processes complex and diverse cloud host scenarios to ensure that the final state of the elastic device meets expectations and improves user experience.

[0167] See also Fig.13 Based on the same inventive concept, the embodiment of the present application further provides a smart network card 130, including:

[0168] The receiving unit 1301 is used to obtain a target number of physical function PF devices required by a target PCIe system to be configured;

[0169] The DP configuration unit 1302 is configured to obtain, based on the target number and an upper limit value of the number of pre-configured downstream port DPs, a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system;

[0170] A topology generating unit 1303 is configured to generate a target system topology corresponding to the target PCIe system based on the first quantity and the second quantity; wherein the second type DP in the target system topology mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices;

[0171] The sending unit 1304 is used to notify the host of the target system topology to instruct the host to configure the target PCIe system according to the target system topology.

[0172] In a possible implementation, the multiple PCI bridges in each bridge group adopt a tree topology structure, the tree topology structure includes N levels of PCI bridges, and each level of PCI bridges includes at least one PCI bridge;

[0173] Wherein, the first-level PCI bridge bridges the PCIe bridge and its corresponding child node in the second-level PCI bridge;

[0174] The i-th level PCI bridge bridges its corresponding parent node in the i-1-th level PCI bridge and its corresponding child node in the i+1-th level PCI bridge, where i is an integer greater than 1 and less than N;

[0175] The Nth level PCI bridge bridges the corresponding parent node in the N-1th level PCI bridge and at least one PF device mounted thereon.

[0176] In a possible implementation manner, the level number N of the tree topology structure is determined based on the total number of PF devices required to be mounted on the corresponding second type DP.

[0177] In a possible implementation manner, a multi-level table lookup unit 1305 is further included, which is used to:

[0178] In response to the table entry update instruction, a system identifier of the target PF device to be stored is obtained; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system;

[0179] Using a preset data encoding method, encode the system identifier to obtain the initial table entry index corresponding to the target PF device;

[0180] If the number of PF devices associated with the initial table entry index has reached the upper limit, then obtain a target table entry index whose number of associated PF devices has not reached the upper limit from the table entry index cascaded with the initial table entry index;

[0181] The target PF device is associated with the target table entry index, and the table entry corresponding to the target table entry index is updated based on the device information of the target PF device.

[0182] In a possible implementation manner, the multi-level table lookup unit 1305 is specifically configured to:

[0183] Traverse the cascaded entry indexes of the initial entry index until a target entry index whose number of associated PF devices does not reach the upper limit is obtained; wherein one traversal includes the following steps:

[0184] Shift the current table entry index to obtain the next-level table entry index corresponding to the current table entry index; wherein, when it is the first traversal, the current table entry index is the initial table entry index, and when it is not the first traversal, the current table entry index is the table entry index obtained by the previous shift processing;

[0185] If the number of PF devices associated with the next-level entry index has reached the upper limit, the next traversal process is entered.

[0186] In a possible implementation manner, the multi-level table lookup unit 1305 is further configured to:

[0187] In response to the device query instruction, obtain the system identification of the target PF device to be queried;

[0188] Using a data encoding method, the system identifier is encoded to obtain a query table item index corresponding to the target PF device;

[0189] Match the target PF device with the PF device associated with the query entry index. If the match succeeds, obtain the device information of the target PF device based on the query entry index.

[0190] If the match fails, the target PF device is matched with the associated PF devices of the table entry index cascaded with the query table entry index in sequence until a successfully matched target table entry index is obtained, and the device information of the target PF device is obtained based on the target table entry index.

[0191] In a possible implementation, the system further includes a state management unit 1306, which is used to:

[0192] Sending the target system topology to a target object corresponding to the target PCIe system, so as to present the target system topology in a terminal device corresponding to the target object;

[0193] receiving a hot-plug control request sent by a target object, where the hot-plug control request is used to request that a target PF device in a target system topology be set to a target hot-plug state;

[0194] A hot-swap control message carrying the system identifier of the target PF device is sent to the host, so that the host switches the target PF device to a target hot-swap state based on the system identifier and the target system topology.

[0195] In a possible implementation, if the target hot-plug state is the inserted state, the state management unit 1306 is specifically configured to:

[0196] When it is determined that the hot-swap state of the target PF device is not plugged in based on the state information stored in the host, the running state of the host is detected;

[0197] If the host is in the power-on state, a hot-plug control message is sent to the host to instruct the hot-plug operation to be performed on the target PF device, and the hot-plug state of the target PF device is updated to the hot-plug in progress state;

[0198] After sending the hot-plug control message carrying the system identifier of the target PF device to the host, the method further includes:

[0199] In response to an indication that the driver of the target PF device is loaded successfully, the hot plug state of the target PF device is updated to an inserted state.

[0200] In a possible implementation, if the target hot-plug state is an unplugged state, the state management unit 1306 is specifically configured to:

[0201] When it is determined that the hot-swap state of the target PF device is inserted based on the state information stored in the host, the running state of the host is detected;

[0202] If the host is in the power-on state, a hot-plug control message is sent to the host to instruct the hot-plug operation to be performed on the target PF device, and the hot-plug state of the target PF device is updated to the hot-plug in progress state;

[0203] After sending the hot-plug control message carrying the system identifier of the target PF device to the host, the method further includes:

[0204] In response to the indication of successful uninstallation of the driver of the target PF device, the hot plug state of the target PF device is updated to a not-inserted state.

[0205] In a possible implementation, the state management unit 1306 is specifically configured to:

[0206] If the detection host is not powered on, the hot-swap state of the target PF device is updated to the target hot-swap state;

[0207] When it is detected that the host is powered on, a state recovery message is sent to the host to instruct the hot-swap state of the target PF device to be restored to the target hot-swap state.

[0208] The device can be used to execute the methods executed by the smart network card in each embodiment of the present application. Therefore, for the functions that can be implemented by each functional module of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0209] See also Fig.14 Based on the same inventive concept, the embodiment of the present application further provides a host 140, including:

[0210] The receiving unit 1401 is configured to receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and multiple PCI bridges mounting PF devices;

[0211] The PCIe system configuration unit 1402 is used to select a PF device that meets the target system topology from the PF devices included in the PCIe system, and respectively call the standard hot-swap controller SHPC of each PF device to initialize the status of each PF device;

[0212] The resource configuration unit 1403 is used to allocate a system identifier to each PF device respectively and return the allocated system identifier to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

[0213] In a possible implementation manner, the PCIe system configuration unit 1402 is further configured to:

[0214] Receiving a hot-plug control message sent by the smart network card, where the hot-plug control message carries a system identifier of the target PF device and is used to instruct to set the target PF device to a target hot-plug state;

[0215] Based on the target system topology and the system identifier, a target SHPC corresponding to the target PF device is determined, and the target SHPC is called to configure the target PF device to the target hot-swap state.

[0216] The device can be used to execute the methods executed by the host in each embodiment of the present application. Therefore, for the functions that can be implemented by each functional module of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0217] See also Fig.15 Based on the same technical concept, the present application also provides a computer device. In one embodiment, the computer device may be the above-mentioned smart network card or host. Fig.15 As shown, it includes a memory 1501 , a communication module 1503 and one or more processors 1502 .

[0218] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0219] The memory 1501 may be a volatile memory, such as a random-access memory (RAM); the memory 1501 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1501 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1501 may be a combination of the above memories.

[0220] The processor 1502 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1502 is used to implement the above-mentioned method for extending the PCIe system when calling the computer program stored in the memory 1501 .

[0221] The communication module 1503 is used to communicate with terminal devices and other servers.

[0222] The specific connection medium between the memory 1501, the communication module 1503 and the processor 1502 is not limited in the embodiment of the present application. Fig.15 In the embodiment, the memory 1501 and the processor 1502 are connected via a bus 1504. The bus 1504 is connected to the processor 1502 via a bus 1504. Fig.15The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1504 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.15 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0223] The memory 1501 stores a computer storage medium, which stores computer executable instructions. The computer executable instructions are used to implement the method for extending the PCIe system of the embodiment of the present application. The processor 1502 is used to execute the method for extending the PCIe system of the above-mentioned embodiments.

[0224] Based on the same inventive concept, an embodiment of the present application also provides a storage medium, which stores a computer program. When the computer program runs on a computer, the computer executes the steps of the method for extending the PCIe system according to various exemplary embodiments of the present application described above in this specification.

[0225] In some possible implementations, various aspects of the method for extending a PCIe system provided in the present application may also be implemented in the form of a computer program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the method for extending a PCIe system according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the steps of each embodiment.

[0226] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0227] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on a computer device. However, the program product of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, and the computer program included therein may be used by or in combination with a command execution system, apparatus or device.

[0228] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0229] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0230] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0231] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0232] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0233] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0234] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0235] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for extending a PCIe system, characterized in that: Applied to a smart network card, the method includes: Obtaining a target number of physical function PF devices required by a target PCIe system to be configured; Based on the target number and an upper limit value of the number of pre-configured downstream port DPs, obtain a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system; Based on the first number and the second number, a target system topology corresponding to the target PCIe system is generated; wherein the second type DP in the target system topology mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices; The target system topology is notified to the host to instruct the host to configure the target PCIe system according to the target system topology.

2. The method according to claim 1, characterized in that The multiple PCI bridges in each bridge group adopt a tree topology structure, wherein the tree topology structure includes N levels of PCI bridges, and each level of PCI bridges includes at least one PCI bridge; Wherein, the first-level PCI bridge bridges the PCIe bridge and its corresponding sub-node in the second-level PCI bridge; The i-th level PCI bridge bridges its corresponding parent node in the i-1-th level PCI bridge and its corresponding child node in the i+1-th level PCI bridge, where i is an integer greater than 1 and less than N; The Nth level PCI bridge bridges the corresponding parent node in the N-1th level PCI bridge and at least one PF device mounted thereon.

3. The method according to claim 2, characterized in that The level number N of the tree topology structure is determined based on the total number of PF devices required to be mounted on the corresponding second type DP.

4. The method according to any one of claims 1 to 3, characterized in that: After sending the system configuration information to the host, the method further includes: In response to the table entry update instruction, obtain a system identifier of the target PF device to be stored; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system; Using a preset data encoding method, encode the system identifier to obtain an initial table entry index corresponding to the target PF device; If the number of PF devices associated with the initial table entry index has reached the upper limit, obtaining a target table entry index whose number of associated PF devices has not reached the upper limit from the table entry index cascaded with the initial table entry index; The target PF device is associated with the target table entry index, and the table entry corresponding to the target table entry index is updated based on the device information of the target PF device.

5. The method according to claim 4, characterized in that Obtaining a target entry index whose number of associated PF devices does not reach an upper limit from the entry index cascaded from the initial entry index includes: Traversing the cascaded table entry index of the initial table entry index until obtaining a target table entry index for which the number of associated PF devices does not reach the upper limit; wherein one traversal includes the following steps: Shifting the current table entry index to obtain the next-level table entry index corresponding to the current table entry index; wherein, when it is the first traversal, the current table entry index is the initial table entry index, and when it is not the first traversal, the current table entry index is the table entry index obtained by the previous shifting process; If the number of PF devices associated with the entry index of the next level has reached the upper limit, the next traversal process is entered.

6. The method according to claim 4, characterized in that After updating the table entry corresponding to the target table entry index based on the device information of the target PF device, the method further includes: In response to the device query instruction, obtain the system identification of the target PF device to be queried; The data encoding method is used to encode the system identifier to obtain the query table item index corresponding to the target PF device; Matching the target PF device with the PF device associated with the query entry index, and if the match succeeds, obtaining device information of the target PF device based on the query entry index; If the match is not successful, the target PF device is matched with the associated PF devices of the table entry index cascaded with the query table entry index in sequence until a successfully matched target table entry index is obtained, and the device information of the target PF device is obtained based on the target table entry index.

7. The method according to any one of claims 1 to 3, characterized in that: After notifying the host of the target system topology, the method further includes: Sending the target system topology to a target object corresponding to the target PCIe system, so as to present the target system topology in a terminal device corresponding to the target object; Receiving a hot-plug control request sent by the target object, wherein the hot-plug control request is used to request setting a target PF device in the target system topology to a target hot-plug state; A hot-plug control message carrying the system identifier of the target PF device is sent to the host, so that the host switches the target PF device to the target hot-plug state based on the system identifier and the target system topology.

8. The method according to claim 7, characterized in that If the target hot-plug state is the inserted state, a hot-plug control message carrying the system identifier of the target PF device is sent to the host, including: When it is determined based on the state information stored in the host that the hot-plug state of the target PF device is an unplugged state, detecting the running state of the host; If the host is in the power-on state, a hot-plug control message is sent to the host to instruct the hot-plug operation to be performed on the target PF device, and the hot-plug state of the target PF device is updated to a hot-plug in progress state; After sending the hot-plug control message carrying the system identifier of the target PF device to the host, the method further includes: In response to an indication that the driver of the target PF device is loaded successfully, the hot plug state of the target PF device is updated to an inserted state.

9. The method according to claim 7, characterized in that If the target hot-plug state is the unplugged state, a hot-plug control message carrying the system identifier of the target PF device is sent to the host, including: When it is determined based on the state information stored in the host that the hot-swap state of the target PF device is the inserted state, detecting the running state of the host; If the host is in the power-on state, a hot-plug control message is sent to the host to instruct to perform a hot-plug operation on the target PF device, and the hot-plug state of the target PF device is updated to a hot-plug in progress state; After sending the hot-plug control message carrying the system identifier of the target PF device to the host, the method further includes: In response to an indication that the driver of the target PF device is successfully uninstalled, the hot plug state of the target PF device is updated to a not-inserted state.

10. The method according to claim 7, characterized in that Sending a hot-swap control message carrying the system identifier of the target PF device to the host includes: If it is detected that the host is in an unpowered state, updating the hot-plug state of the target PF device to the target hot-plug state; When it is detected that the host has been powered on, a state recovery message is sent to the host to instruct the hot-plug state of the target PF device to be restored to the target hot-plug state.

11. A method for extending a PCIe system, characterized in that: Applied to a host, the method comprises: Receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts multiple PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and multiple PCI bridges mounting PF devices; Selecting a PF device that meets the target system topology from the PF devices included in the system, and calling the standard hot-swap controller SHPC of each PF device to initialize the state of each PF device; A system identifier is allocated to each PF device respectively, and the allocated system identifier is returned to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

12. The method according to claim 11, characterized in that After allocating system identifiers to the respective PF devices and returning the allocated system identifiers to the smart network card, the method further includes: Receiving a hot-plug control message sent by the smart network card, where the hot-plug control message carries a system identifier of the target PF device and is used to instruct to set the target PF device to a target hot-plug state; Based on the target system topology and the system identifier, a target SHPC corresponding to the target PF device is determined, and the target SHPC is called to configure the target PF device to the target hot-swap state.

13. A smart network card, characterized in that: include: A receiving unit, configured to obtain a target number of physical function PF devices required by a target PCIe system to be configured; A DP configuration unit, configured to obtain, based on the target number and an upper limit value of the number of pre-configured downstream port DPs, a first number of first-type DPs for mounting a single PF device and a second number of second-type DPs for mounting multiple PF devices in the target PCIe system; A topology generating unit, configured to generate a target system topology corresponding to the target PCIe system based on the first quantity and the second quantity; wherein the second type DP in the target system topology mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices; The sending unit is used to notify the host of the target system topology to instruct the host to configure the target PCIe system according to the target system topology.

14. A host, characterized in that: include: A receiving unit, configured to receive a target system topology sent by the smart network card; wherein, in the target system topology, each second type DP mounts a plurality of PF devices through its corresponding bridge group, and the bridge group includes a PCIe bridge bridging the second type DP and a plurality of PCI bridges mounting the PF devices; A PCIe system configuration unit, configured to select a PF device that meets the target system topology from the PF devices included in the PCIe system, and to call the standard hot-swap controller SHPC of each PF device to initialize the state of each PF device; A resource configuration unit is used to allocate a system identifier to each PF device respectively and return the allocated system identifier to the smart network card; wherein the system identifier is generated based on the target system topology and is a unique identifier of the target PF device in the target PCIe system.

15. A cloud server, characterized in that: It comprises the smart network card as claimed in claim 13 and the host as claimed in claim 14.

16. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 or 11 to 12 are implemented.

17. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 or 11 to 12 are implemented.

Citation Information

Patent Citations

  • PCIe Switch system extension management method

    CN113515478A

  • PCIe system expansion method, PCIe switching device and PCIe system

    CN114006875A

  • PCI-E (Peripheral Component Interconnect Express) topology method and device, equipment and storage medium

    CN115114219A

  • Self-adaption method for rapid peripheral component interconnection bus resources of data processing unit

    CN116095023A

  • Emulated switch with hot-plugging

    US10509758B1