PCIe device remote control method, device, electronic device, and medium

Abstract

The present invention discloses a PCIe device remote control method, device, electronic device and medium. The device comprises: a local server and a remote server; a PCIe hardware device control component, a virtual PCIe bus and a plurality of local RDMA network cards are deployed on the local server; a PCIe hardware device, a PCIe hardware device proxy component and a plurality of remote RDMA network cards are deployed on the remote server; the PCIe hardware device control component is used to realize the virtual PCIe device presentation of the PCIe hardware device deployed on the remote server on the local CPU; the virtual PCIe bus is used to manage the virtual PCIe bus and the virtual PCIe devices on the bus for the local CPU; the PCIe hardware device proxy component is paired with the PCIe hardware device deployed on the remote server one by one, and is used to complete the proxy forwarding of the PCIe request between the remote RDMA network card and the PCIe hardware device.

Description

Technical Field

[0001] The present invention belongs to the technical field of PCIe bus and equipment, and in particular, relates to a PCIe equipment remote control method, device, electronic equipment, and medium. Background Art

[0002] With the rapid development of information technology, the performance requirements of computer systems are constantly increasing. PCI Express (PCIe), as the current mainstream high-speed serial computer expansion bus interface standard, has been widely used in various computer systems. With its high speed, high bandwidth and low latency, PCIe has become the preferred interface for connecting various high-performance peripherals (such as graphics processors, network interface cards and storage devices). However, with the complexity of system architecture and the diversification of usage scenarios, the limited physical distance of PCIe devices has become an important bottleneck for its application expansion.

[0003] The traditional PCIe bus design is based on point-to-point connection. The communication between the device and the host must be carried out within a limited physical distance, which is usually within the server device. This limitation makes it impossible to deploy PCIe devices in long distances or dispersed locations in some scenarios, especially in application areas such as data centers, remote computing, and industrial automation, and it is also impossible to flexibly deploy and use the required PCIe devices. In this case, extension technology came into being to solve the problem of physical distance limitations, thereby realizing remote connection and control of PCIe devices.

[0004] At present, a variety of PCIe remote technologies have been proposed, for example, using the RDMA network to forward and synchronize the middleware software requests between the user end and the real PCIe device end, or expanding and dynamically allocating PCIe devices based on the PCIe network. In the solution based on the RDMA network for request forwarding and synchronization, the software user end adds a middleware that adapts the application and performs request forwarding processing, and the matching new middleware on the real PCIe device end receives the request and calls the real driver of the PCIe device to send the request to the PCIe device and return the return data of the PCIe physical device; in the solution based on the PCIe network expansion and allocation of PCIe devices, a PCIe bridge type PCIe device is added to the software user end and the real PCIe device end respectively, and the PCIe bridge is used to forward the PCIe request.

[0005] However, the existing PCIe remote technology also has many shortcomings and challenges. For the software-based request forwarding solution, the middleware needs to hijack and forward higher-level device requests, which also limits the application software to only use higher-level requests for PCIe devices and cannot directly perform underlying operations. At the same time, the middleware needs to be upgraded synchronously with the upgrade of the PCIe device native software interface version. In addition, the request processing process needs to be converted multiple times between kernel mode and user mode, which affects the overall request response performance. For PCIe network expansion solutions, the extension distance is limited by the length of the PCIe bus, and the PCIe expansion range is usually limited to the cabinet level. In addition, the number of PCIe switching expansion device interfaces is usually very limited. The overall solution has great restrictions on large-scale PCIe device remote reuse scenarios. At the same time, the relatively high prices of PCIe devices and components are not conducive to the construction of large-scale remote clusters.

[0006] In summary, although the existing PCIe remote technology has solved the problem of limited physical distance of PCIe devices to a certain extent, it still faces a series of challenges such as high cost, low performance, high power consumption, and poor scalability. Summary of the invention

[0007] In view of the deficiencies in the prior art, embodiments of the present invention provide a method, apparatus, electronic device, and medium for remotely controlling a PCIe device.

[0008] In a first aspect, an embodiment of the present invention provides a PCIe device remote control apparatus, the apparatus comprising: a local server and a remote server;

[0009] Wherein, the local server is deployed with a PCIe hardware device control component, a virtual PCIe bus, and several local RDMA network cards;

[0010] The remote server is deployed with a PCIe hardware device, a PCIe hardware device proxy component, and several remote RDMA network cards;

[0011] The PCIe hardware device control component is used to realize the virtual PCIe device presentation of the PCIe hardware device deployed on the remote server on the local CPU; the virtual PCIe bus is used to manage the virtual PCIe bus and the virtual PCIe devices on the bus for the local CPU;

[0012] The PCIe hardware device proxy component is paired one-to-one with the PCIe hardware device deployed on the remote server, and is used to complete the proxy forwarding of the PCIe request between the remote RDMA network card and the PCIe hardware device.

[0013] In a second aspect, an embodiment of the present invention provides a PCIe device remote control method. In an initialization phase, the method includes:

[0014] Starting the local server, and pre-allocating the address space of each virtual PCIe device in the PCIe hardware device control component;

[0015] When the local server accesses the remote PCIe hardware device, determine the remote server that needs to provide the remote PCIe hardware device, determine the remote PCIe hardware device and the remote PCIe hardware device to be accessed, and remove the remote PCIe hardware device to be accessed from the logical layer of the remote server;

[0016] Configure the PCIe hardware device proxy component corresponding to the remote PCIe hardware device, and set the working mode of the PCIe hardware device proxy component to the remote service mode;

[0017] Allocate a PCIe BAR space address range in the PCIe hardware device control component for the virtual PCIe device;

[0018] Obtain the PCIe BAR space requirement corresponding to the remote PCIe hardware device, and reallocate the PCIe BAR space address corresponding to the remote PCIe hardware device according to the PCIe BAR space start address corresponding to the virtual PCIe device in the PCIe hardware device control component, so that the PCIe BAR space address of the remote PCIe hardware device is the same as the PCIe BAR space address of the virtual PCIe device in the PCIe hardware device control component; and synchronize the PCIe BAR space address of the remote PCIe hardware device to the corresponding PCIe hardware device proxy component;

[0019] Establishing virtual address information for address spaces related to the PCIe hardware device in the PCIe hardware device control component and the PCIe hardware device proxy component;

[0020] The address information of the virtual PCIe device in the PCIe hardware device control component is exchanged with the address information of the PCIe hardware device proxy component to obtain the complete address information; and the complete address information is registered in the memory of the local RDMA network card and the remote RDMA network card respectively to establish a connection between the local RDMA network card and the remote RDMA network card;

[0021] Configure a virtual PCIe bus and add a sub-bus under the virtual PCIe root bus. The sub-bus corresponds to the virtual PCIe device in the PCIe hardware device control component.

[0022] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned PCIe device remote control method.

[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the above-mentioned PCIe device remote control method when executed by a processor.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention provides a PCIe device remote control method and device, which supports remote virtual access operation and use of PCIe devices by analyzing and processing PCIe messages, and improves the distance and scale of PCIe devices through RDMA networks. The present invention can improve transmission efficiency while extending the connection distance of PCIe devices, and achieve good compatibility and standardization, providing more complete and efficient solutions for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0027] Figure 1 A schematic diagram of remote control of a PCIe device provided by an embodiment of the present invention;

[0028] Figure 2 A structural diagram of a PCIe device remote control provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the composition of address segments in a PCIe hardware device control component provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the composition of address segments in a PCIe hardware device proxy component provided in an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0034] like Figure 1 As shown, an embodiment of the present invention provides a PCIe device remote control device, the device comprising: a local server and a remote server;

[0035] The local server is deployed with a PCIe hardware device control component (DEV Master), a virtual PCIe bus (Virt PCIe), a local PCIe device management module (DEV management Local), and several local RDMA network cards.

[0036] The remote server is deployed with a PCIe hardware device (PCIe DEV), a PCIe hardware device agent component (DEVAgent), a remote PCIe device management module (DEV management Remote), and several remote RDMA network cards.

[0037] It should be noted that several PCIe hardware devices deployed on a remote server can be mapped to a local server for virtual PCIe device configuration; or, several PCIe hardware devices deployed on multiple remote servers can be mapped to a local server for virtual PCIe device configuration; or, several PCIe hardware devices deployed on multiple remote servers can be mapped to multiple local servers for virtual PCIe device configuration; in this example, the number of local servers and remote servers is not limited.

[0038] The remote PCIe hardware device presents a virtual PCIe device in the local device CPU. The local CPU reads and writes the virtual PCIe device as direct reads and writes to the PCIe hardware device control component (DEV Master). The local RDMA network card, remote RDMA network card and PCIe hardware device agent component (DEV Agent) implement the reading and writing of the actual physical PCIe hardware device.

[0039] The remote physical PCIe hardware device reads and writes the local CPU, memory, and devices. The PCIe hardware device agent component (DEV Agent), the remote RDMA network card, and the local RDMA network card forward the request to the local server, and the PCIe hardware device control component (DEV Master) or directly the local RDMA network card implements the reading and writing of the local CPU, memory, and devices.

[0040] Furthermore, the PCIe hardware device control component (DEV Master) is a hardware PCIe device for adapting to the local CPU. As a hardware device form, the PCIe hardware device control component (DEV Master) implements the virtual PCIe DEV device presentation of one or more PCIe hardware devices (PCIe DEV) deployed on the remote server on the local CPU, so as to enable direct read and write access between the local CPU and the corresponding remote PCIe hardware device PCIe DEV.

[0041] The PCIe hardware device control component (DEV Master) can support multiple virtual PCIe devices to be virtually presented on the local CPU, and each virtual PCIe device is associated with a remote PCIe hardware device agent component (DEV Agent); a DEV Master can correspond to multiple local RDMA network cards and multiple remote PCIe hardware devices, and mainly completes the processing of PCIe requests (configuration requests and read and write requests) and RDMA transmission control; the PCIe hardware device control component (DEV Master) supports interaction with the local RDMA network card, and generates RDMA read and write requests or local read and write requests according to data read and write request information.

[0042] like Figure 3 and Figure 4 As shown, each virtual PCIe device in the PCIe hardware device control component (DEV Master) and the PCIe hardware device agent component (DEV Agent) contains four address ranges: a BAR address range, a CFG configuration address range, a CMD command address range for assisting communication with a remote PCIe hardware device, and a Trans transmission address range for assisting data transmission with a remote PCIe hardware device.

[0043] Among them, the BAR address segment size in the PCIe hardware device control component (DEV Master) virtual device is the same as the BAR address segment size in the PCIe hardware device agent component (DEV Agent), and the BAR address segment in the PCIe hardware device control component (DEV Master) virtual device is accessed by the local CPU; the CFG configuration address segment size in the PCIe hardware device control component (DEV Master) virtual device is twice the CFG configuration address segment size in the PCIe hardware device agent component (DEV Agent), and the CFG configuration address segment in the PCIe hardware device control component (DEV Master) virtual device is divided into two identical segments, which are used for CPU access and local RDMA network card access respectively; the CMD command address segment size in the PCIe hardware device control component (DEV Master) virtual device is the same as the CMD command address segment size in the PCIe hardware device agent component (DEV Agent); the Trans transmission segment in the PCIe hardware device control component (DEV Master) virtual device and the Trans transmission segment address in the PCIe hardware device agent component (DEV Agent) are used for auxiliary data transmission, and their sizes are not strongly associated.

[0044] It should be noted that the address interval in the BAR address space segment can be mapped one-to-one with the actual PCIe BAR address interval of the remote physical PCIe hardware device; the local CPU completes the PCIe configuration read and write operations on the remote physical PCIe hardware device by reading and writing the above-mentioned configuration space segment, and the read and write operations on the BAR space of the remote physical PCIe hardware device will be completed through the above-mentioned BAR address space segment; the DEV Master reads and writes the configuration space segment and BAR address space segment of the remote PCIe hardware device, which is realized by the RDMA network card's read and write request to the DEV Agent's PCIe configuration address space segment and BAR address space segment; the physical PCIe hardware device's read and write requests to the local CPU and other device spaces are realized by the DEV Agent interacting with the DEV Master through its command CMD space segment and transmission Trans segment.

[0045] During initialization, the PCIe hardware device control component (DEV Master) applies for a sufficiently large PCIe BAR space. When a remote PCIe hardware device is subsequently added, the DEV Master allocates an address space in its own BAR space as the dedicated address interval for the remote PCIe hardware device. The address interval of each remote PCIe hardware device in the DEV Master includes the configuration address segment space, BAR address segment space, CMD command address segment space, and Trans address segment space. The DEV Master provides an independent address segment space for each remote physical PCIe DEV, which is nested in the PCIe BAR address of the DEVMaster to respond to and process the PCIe read and write request messages from the CPU and RDMA network card to the address space.

[0046] Furthermore, the process of the PCIe hardware device control component (DEV Master) interacting with the local RDMA network card to generate an RDMA read / write request or a local read / write request includes:

[0047] The local CPU reads and writes the configuration of the remote physical PCIe hardware device and the PCIe BAR space data, which are converted by the DEVMaster into RDMA read and write requests to the remote corresponding DEV Agent;

[0048] The remote PCIe hardware device transmits the read and write requests for the local address space to the local DEVMaster through RDMA read and write requests. The DEV Master responds to the read and write requests and configures the software and hardware environment of the local server to assist in completing the read and write requests.

[0049] The address segments corresponding to the virtual Virt devices in DEV Master are the BAR address segment interval, the CFG configuration address segment interval, the CMD command address segment interval for assisting communication with remote PCIe hardware devices, and the Trans transmission address segment interval for assisting data transmission with remote PCIe hardware devices, which have been pre-registered in the local RDMA network card memory;

[0050] The DEV Master has pre-established a connection with the corresponding remote RDMA network card; the DEV Master interacts with the local RDMA network card to generate an RDMA request, and submits the transmission task to the local RDMA network card. The specific method may include but is not limited to the DEV Master directly generating and submitting to the local RDMA network card, and submitting to the local RDMA network card with the help of auxiliary software.

[0051] The process in which the CPU's configuration space read and write request for the remote PCIe hardware device is forwarded by the DEV Master includes:

[0052] The configuration space address segment of each virtual Virt device is divided into two parts of the same size, which are used for reading and writing with the CPU and the local RDMA network card respectively. The two parts of the space are mapped one by one to each other, and DEVMaster completes the data forwarding between the two parts of the space;

[0053] When the CPU sends a read / write request to the CPU portion of the DEV Master configuration space address segment, the DEV Master stores the read / write request and data, and then the local RDMA network card reads and writes to the corresponding address of the RDMA portion in the configuration space segment to transmit the data to the remote end;

[0054] When the CPU makes a PCIe read request to the DEV Master configuration space address segment, after receiving the data written by the local RDMA network card to the address corresponding to the configuration space segment, the DEV Master returns the data to the CPU in the form of a PCIe read completion message. The CPU receives the PCIe read completion message to indicate the end of the configuration read;

[0055] When the CPU makes a PCIe write request to the DEV Master configuration space address segment, the CPU software waits for the DEV Master to send a signal that the remote PCIe hardware device configuration write is completed after the write operation is completed; after receiving the RDMA network card's RDMA write signal from the DEV Master's configuration space address segment to the remote DEV Agent's configuration space address segment, the DEV Master sends a signal to the CPU that the remote PCIe hardware device configuration write is completed. The DEV Master sends the CPU a signal that the write is completed, including but not limited to writing a flag register value and sending an interrupt signal.

[0056] The process in which the CPU's BAR address space read and write request to the remote PCIe hardware device is forwarded by the DEV Master includes:

[0057] The direct address of the PCIe read and write instructions issued by the local CPU is the independent address space of the remote physical PCIe hardware device in the BAR space of the DEV Master, specifically pointing to the BAR address space segment therein;

[0058] The DEV Master maps the local BAR space address of the CPU read / write request to determine the address in the corresponding BAR space address segment of the remote DEV Agent, and allocates the address interval in the DEV Master's Trans address space segment for this data transmission. The BAR space address of the remote DEV Agent and the locally allocated Trans address are used as the source and destination addresses of the local RDMA transmission.

[0059] When the local RDMA network card sends a PCIe read / write data request to the above-mentioned Trans address, the DEV Master obtains data from the aforementioned CPU write instruction and forwards it to the local RDMA network card according to the mapping relationship when the Trans address is allocated, or uses the local RDMA network card to write the request data to return a PCIe read completion message to the CPU read instruction.

[0060] The process in which the local DEV Master responds to the remote PCIe hardware device's read and write requests for the local address space and completes the necessary software and hardware configuration includes:

[0061] The data transmission modes when the DEV Master responds to the read and write requests of the remote DEV Agent include the DEV Master read and write forwarding local data mode and the RDMA network card directly reading and writing local data mode;

[0062] In the above-mentioned DEV Master forwarding mode, after receiving the read / write request containing the final destination address from the DEV Agent, the DEV Master allocates the corresponding data read / write address interval in the transmission Trans address segment space, and then notifies the DEVAgent to perform RDMA read / write on the Trans segment address interval allocated in the DEV Master. After receiving the PCIe read / write on the Trans address interval, the DEV Master converts the read / write into the read / write on the final destination address in the local address space;

[0063] In the direct read and write mode of the RDMA network card, when the DEV Master receives the read and write request containing the final destination address from the DEV Agent, the DEV Master establishes the corresponding virtual address space page table for the final read and write address range, completes the registration of the address range to the RDMA network card, and then notifies the DEV Agent to directly perform RDMA read and write on this final destination address.

[0064] Among them, the configuration reading and writing of the physical PCIe hardware device by the DEV Master is realized by reading and writing the PCIe configuration address space segment in the DEV Agent through the RDMA network card. The process mainly includes:

[0065] When the CPU sends a configuration read / write request to the DEV Agent through the DEV Master, the remote RDMA network card performs PCIe read / write on the PCIe configuration address space segment of the DEV Agent, and the DEV Agent sends the read / write request to the physical DEV device in the form of a PCIe configuration message and receives the corresponding return information;

[0066] When the remote RDMA network card sends a PCIe write request to the configuration space segment of the DEV Agent, the DEV Agent determines the configuration space address and write data content of the PCIe DEV device of the current write operation based on the information of the write request, generates a PCIe configuration write command for the PCIe DEV device containing the corresponding configuration address and data, and sends it to the PCIe DEV device. After receiving the response information of the configuration write completion of the PCIe DEV device, the DEV Agent sends a response information to the local DEV Master through the command CMD address space segment to notify the DEV Master that the write request has been completed;

[0067] When the remote RDMA network card sends a PCIe read request to the configuration space segment of the DEV Agent, the DEV Agent determines the configuration space address of the PCIe hardware device of the current read operation according to the information of the read request (the configuration space address can be determined in a way that realizes a one-to-one mapping between the write information received by the DEV Agent and the PCIe DEV configuration address, and the specific method includes but is not limited to a one-to-one mapping between the configuration request address interval of the DEV Agent and the PCIe DEV configuration space, and the write data contains the PCIe DEV configuration address), generates a PCIe configuration read command for the PCIe hardware device containing the configuration space read address, and sends it to the PCIe hardware device. After receiving the response information of the configuration read completion of the PCIe hardware device, the DEV Agent generates a response PCIe message to the remote RDMA network card read request according to the configuration read data, and the RDMA network card completes the transmission of the read completion information to the local DEV Master.

[0068] The local DEV Master reads and writes the BAR address space of the remote physical PCIe hardware device through the RDMA network card to the BAR address space of the DEV Agent. The process includes:

[0069] When the DEV Agent receives a PCIe read / write request from the RDMA network card for its BAR address segment space, the DEV Agent determines the BAR address of the physical PCIe DEV device that actually needs to be accessed based on the requested address, and sends the corresponding PCIe read / write request message to the PCIe DEV device; if it is a read request, the DEV Agent returns the corresponding PCIe read response message to the RDMA network card after receiving the read request response from the physical PCIe DEV device.

[0070] The read and write requests from the physical PCIe hardware device to the local CPU and other device spaces are forwarded by the DEV Agent. The process includes:

[0071] The DEV Agent receives a PCIe read / write request for a certain address sent by the physical PCIe DEV device, allocates the address range of this RDMA transmission in the transmission Trans space segment of the DEV Agent and selects the data transmission mode (the transmission mode refers to the DEV Master forwarding mode or the RDMA direct read / write mode), and then notifies the local DEV Master of the destination address, data length, and transmission type of the transmission request. The DEV Master allocates the transmission Trans space segment or establishes a virtual page table of the local physical address according to different transmission modes; then, data is transmitted between the remote DEV Agent and the local through RDMA (the initiator of the actual RDMA request is not restricted, and the RDMA non-initiator is required to synchronize the local RDMA actual transmission address information to the RDMA initiator). When the DEV Agent receives a read / write request from the RDMA network card, it responds to the read request of the RDMA network card with the PCIe write data sent by the physical PCIe DEV device received before, or responds to the read request of the physical PCIe DEV device with the write data of the RDMA network card.

[0072] Furthermore, the virtual PCIe bus (Virt PCIe) is a software module used to manage a virtual PCIe bus and virtual PCIe DEV devices on the bus for a local CPU.

[0073] Specifically, the virtual bus of the virtual PCIe bus (Virt PCIe) includes a root bus and multiple sub-buses. The sub-buses are mounted under the root bus, and each virtual PCIe device is mounted under a sub-bus. There is a one-to-one correspondence between the sub-buses and the virtual PCIe devices, and the number of sub-buses increases and decreases synchronously with the addition and deletion of virtual PCIe devices; the configuration reading and writing of the devices on its bus through the sub-bus is implemented as the reading and writing of the CFG configuration space address interval corresponding to the virtual PCIe device in the PCIe hardware device control component (DEV Master).

[0074] Furthermore, the PCIe hardware device agent component (DEV Agent) is used to adapt to the PCIe hardware device (PCIe DEV) deployed on the remote server. As a hardware device form, the PCIe hardware device agent component (DEV Agent) logically presents a single-port PCIe bridge, which is responsible for the external adaptation and management of the remote PCIe DEV device at a macro level. The DEVAgent and the PCIe DEV device are in a one-to-one pairing relationship. In terms of physical connection, it is the only hardware device that the PCIe DEV directly connects to. It mainly completes the forwarding of PCIe requests between the remote RDMA network card and the PCIe DEV (configuration access requests and BAR space read and write requests), and at the same time realizes the logical isolation of the PCIe DEV hardware from other devices on the remote server.

[0075] Furthermore, the PCIe hardware device agent component (DEV Agent) has two main working modes: transparent bridge mode and remote service mode; in the remote service mode, all external requests of PCIe DEV are processed by DEV Agent as requests to the local server; the PCIe BAR address space of DEV Agent contains the control address segment space of PCIe DEV device, and the control address segment space contains the configuration address segment space, BAR address segment space, CMD command address segment space and transmission Trans address segment space.

[0076] Specifically, the PCIe hardware device agent component (DEV Agent) is a PCIe NTB bridge in hardware structure, one end of which is connected to a remote PCIe hardware device and the other end is connected to the PCIe bus of the remote server; in terms of logical function, it provides forwarding processing of PCIe read and write requests between the remote PCIe hardware device and the remote RDMA network card; in the PCIe bus of the remote server, an independent PCIe BAR address space is provided for the corresponding physical PCIe hardware device, and external PCIe address reads and writes to the address space are converted into reads and writes to the physical PCIe hardware device; the PCIe hardware device agent component (DEV Agent) appears as an ordinary EP device to the remote server PCIe bus and as a quasi-RC device (i.e., root complex) to the physical PCIe hardware device.

[0077] Among them, DEV Agent presents the physical PCIe hardware device as a quasi-RC device, specifically: it can support actively initiating PCIe configuration read and write messages for the physical PCIe DEV, and can support forwarding PCIe interrupt and event messages to the local DEVMaster.

[0078] Furthermore, the local PCIe device management module (DEV management Local) and the remote PCIe device management module (DEV management Remote) are both software modules; the local PCIe device management module is used to manage virtual PCIe devices, configure and manage PCIe hardware device control components (DEV Master), virtual PCIe buses (VirtPCIe), and assist in RDMA transmission; the remote PCIe device management module (DEV management Remote) is used to manage the PCIe hardware devices (PCIe DEV) of the cluster and control the remote use allocation of PCIe hardware devices, configure and manage PCIe hardware device agent components (DEV Agent), and assist in RDMA transmission, such as RDMA network card command generation, memory page table management of hardware address space, etc.

[0079] Furthermore, the local RDMA network card and the remote RDMA network card are both standard RDMA network card hardware devices and supporting software, supporting RDMA data transmission.

[0080] It should be noted that the core idea of ​​the present invention is to realize remote direct reading and writing of hardware through the RDMA network, and to set up PCIe message processing devices locally and remotely to directly access and process data on the PCIe devices inside the server, so as to realize the connection between the RDMA network card and the PCIe terminal. The present invention introduces the PCIe hardware device agent component (DEV Agent), the PCIe hardware device control component (DEV Master) and the RDMA network card (RNIC), so as to realize the flexible virtual presentation of the PCIe hardware devices in the remote server device in the local server, so that the software in the local server continues to reuse the driver and library software of the native PCIe device. In the scheme of the present invention, the local server can arbitrarily select a PCIe card in a remote server and map it to the local server for use as a virtual device. At the same time, the local server can flexibly select the network card for realizing data synchronization transmission according to its own network card hardware resources and performance requirements. In the scheme of the present invention, the software in the local server can support memory access and read and write, configuration space read and write of the remote PCIe device, and can also support the remote PCIe device's read and write access to the local server memory and interrupt sending, and can also support P2P transmission between remote PCIe devices corresponding to two virtual PCIe devices.

[0081] like Figure 2As shown, in this example, the access to the PCIe device is mainly memory and configuration space reading and writing. In addition, GPU DMA, MSI / MSI-X interrupts, and P2P transmission characteristics are composed of configuration space reading and writing and memory reading and writing processes at the hardware operation level. That is, the support of typical configuration access and memory access processes can support the use of typical functional characteristics of PCIe devices. The following will describe the processing flow under system initialization, PCIe configuration reading and writing, and ordinary memory reading and writing operations respectively.

[0082] Furthermore, an embodiment of the present invention provides a PCIe device remote control method. In the initialization stage, the method comprises the following steps:

[0083] Step S1, start the local server and pre-allocate the address space of each virtual PCIe device in the PCIe hardware device control component (DEV Master);

[0084] Step S2, when the local server accesses the remote PCIe hardware device, the remote server that needs to provide the remote PCIe hardware device is determined, the remote PCIe hardware device management module (DEV management Remote) determines the remote PCIe hardware device and the remote PCIe hardware device to be accessed, and removes the remote PCIe hardware device to be accessed from the logical layer of the remote server;

[0085] Step S3, the remote PCIe hardware device management module configures a PCIe hardware device agent component (DEV Agent) corresponding to the remote PCIe hardware device, and sets the working mode of the PCIe hardware device agent component to a remote service mode.

[0086] Step S4, the local PCIe hardware device management module (DEV management Local) allocates a PCIe BAR space address range in the PCIe hardware device control component to the virtual PCIe device;

[0087] Step S5, the remote PCIe hardware device management module obtains the PCIe BAR space requirement corresponding to the remote PCIe hardware device, and reallocates the PCIe BAR space address corresponding to the remote PCIe hardware device according to the PCIe BAR space start address corresponding to the virtual PCIe device in the PCIe hardware device control component, so that the PCIe BAR space address of the remote PCIe hardware device is the same as the PCIe BAR space address of the virtual PCIe device in the PCIe hardware device control component; and synchronizes the PCIe BAR space address of the remote PCIe hardware device to the corresponding PCIe hardware device proxy component;

[0088] Step S6: establishing virtual address information for the address space related to the PCIe hardware device in the PCIe hardware device control component and the PCIe hardware device proxy component through the remote PCIe hardware device management module and the local PCIe hardware device management module.

[0089] Step S7, exchanging the address information of the virtual PCIe device in the PCIe hardware device control component with the address information of the PCIe hardware device proxy component to obtain complete address information; and registering the complete address information to the memory of the local RDMA network card and the remote RDMA network card respectively, to establish a connection between the local RDMA network card and the remote RDMA network card;

[0090] Further, the address information of the virtual PCIe device includes PCIe DEV BAR space, DEV Config configuration space, DEV CMD command space, and DEV Transmit transmission space; the address information of the PCIe hardware device proxy component includes its physical address and virtual address;

[0091] Step S8, the local PCIe hardware device management module configures a virtual PCIe bus (Virt PCIe), adds a sub-bus under the virtual PCIe root bus, and the sub-bus corresponds to the virtual PCIe device in the PCIe hardware device control component; sets the configuration access address of the virtual PCIe device under the sub-bus to the Config configuration space in the virtual PCIe device address, and adds the virtual PCIe device in the local PCIe hardware device management module.

[0092] Furthermore, an embodiment of the present invention provides a method for remotely controlling a PCIe device, in a PCIe configuration read and write phase, the method comprising the following steps:

[0093] Step S1, calling the virtual PCIe bus configuration read and write interface, sending a configuration read and write request to the corresponding address in the Config configuration space where the virtual PCIe device in the PCIe hardware device control component is located; the read and write request includes the device number, PCIe function number, configuration access register address, read and write type, and if it is a write request, it also needs to include the data information to be written;

[0094] Step S2, when the PCIe hardware device control component receives a configuration read / write request, it encapsulates the configuration read / write request into an RDMA write data request, communicates with the remote RDMA network card through the local RDMA network card, and sends the RDMA write data request to the config configuration space of the PCIe hardware device proxy component; sends an RDMA read data request according to the result data address in the config configuration space of the PCIe hardware device proxy component; wherein the destination address of the RDMA read data request is the storage address of the result data in the config configuration space in the PCIe hardware device control component;

[0095] Step S3, when the PCIe hardware device proxy component receives a configuration read / write request transmitted by a remote RDMA network card, based on the device number, PCIe function number, and destination register information, the write request also needs to include data information, and generates and sends a PCIe configuration read / write message to the PCIe hardware device, and configures the bus number in the PCIe configuration read / write message to be the bus number of the physical PCIe bus where the PCIe hardware device is located;

[0096] Step S4, after receiving the configuration read / write completion response message returned by the PCIe hardware device, the PCIe hardware device proxy component places the configuration read / write result in the result data access address of its configuration space; under the configuration read request condition, the configuration space register value also needs to be synchronously placed in the result data access address;

[0097] Step S5, when the PCIe hardware device proxy component receives a PCIe read request for configuration result data sent by the remote RDMA network card, the read / write result data is returned to the remote RDMA network card in the form of a PCIe read / write response message; when the PCIe hardware device proxy component does not receive a PCIe read request for configuration result data from the remote RDMA network card, it waits for the request to arrive, and then returns the configuration read / write result data to the remote RDMA network card;

[0098] Step S6, when the PCIe hardware device control component receives a PCIe write data request from the local RDMA network card to the result data storage address in its config configuration space, it calls the configuration read and write interface of the virtual PCIe bus; when the configuration read and write is completed, it returns the configuration write request or returns the configuration read request together with the configuration data.

[0099] Furthermore, an embodiment of the present invention provides a PCIe device remote control method, in the memory read and write access phase from the local server to the remote server, the method includes:

[0100] Step S1, since the PCIe BAR address of the virtual PCIe device is the PCIe BAR address segment of the virtual PCIe device corresponding to the PCIe hardware device control component (DEVMaster), when the local CPU performs memory read and write to the PCIe BAR address of the virtual PCIe device, a memory read and write request is sent to the PCIe BAR address of the virtual PCIe device corresponding to the PCIe hardware device control component.

[0101] Step S2, when the PCIe hardware device control component receives a PCIe memory read / write request for the first address A (if it is a PCIe memory write request, the PCIe hardware device control component needs to cache the data to be written), according to the mapping relationship between the local BAR address synchronously determined in the initialization phase and the BAR address of the PCIe hardware device proxy component, obtain the second address B corresponding to the first address A for the PCIe hardware device proxy component to receive the memory read / write request;

[0102] Submit an RDMA read / write request, and perform RDMA transmission through the local RDMA network card and the remote RDMA network card; the RDMA read / write request is consistent with the memory read / write request type received by the PCIe hardware device control component; the source address and destination address of the RDMA read / write request are the local first address and the remote second address respectively; it should be noted that since the physical address and virtual address mapping have been established during initialization and are synchronized between the PCIe hardware device control component and the PCIe hardware device proxy component, the distinction between virtual address and physical address is not shown here.

[0103] Step S3A, when the local CPU issues a memory write request, after the local server and the remote server start the RDMA request, the local RDMA network card will send a PCIe read data request to the first address A in the PCIe hardware device control component. After receiving the PCIe read data request sent by the local RDMA network card, the PCIe hardware device control component returns the cached data to be written as the data in the PCIe read completion message to the local RDMA network card, and the local RDMA network card sends the write request and the data to be written to the remote RDMA network card;

[0104] After the remote RDMA network card receives the write request and the data to be written transmitted by the local RDMA network card, the remote RDMA network card sends a PCIe write request to the second address B of the PCIe hardware device proxy component; after the PCIe hardware device proxy component receives the PCIe write request sent by the remote RDMA network card, it caches the data to be written, calculates the corresponding third address C in the PCIe hardware device according to the second address B, and sends a PCIe write request message to the PCIe hardware device; wherein, the destination address of the PCIe write request message is the third address C, and the data of the PCIe write request message is the cached data to be written.

[0105] Step S3B, when the local CPU issues a memory read request, after the local server and the remote server start the RDMA request, the local RDMA network card receives the read data returned by the PCIe hardware device proxy component sent by the remote RDMA network card, and the local RDMA network card sends a PCIe write data request to the first address A in the PCIe hardware device control component. After receiving the PCIe write data request, the PCIe hardware device control component caches the write request data and returns the write request data to the local CPU as the data in the completion message of the local CPU's PCIe read data request for the first address; the local RDMA network card sends a read request to the remote RDMA network card;

[0106] The remote RDMA network card will receive the read request transmitted by the local RDMA network card, and send a PCIe read request to the second address B of the PCIe hardware device proxy component; after receiving the PCIe read request sent by the remote RDMA network card, the PCIe hardware device proxy component calculates the corresponding third address C in the PCIe hardware device according to the second address B, and sends a PCIe read request message to the PCIe hardware device; wherein the destination address of the PCIe read request message is the third address C; after receiving the PCIe read response message returned by the PCIe hardware device, the PCIe hardware device proxy component extracts the data in the PCIe read response message, and returns the PCIe read response message and data to the read request of the remote RDMA network card.

[0107] Furthermore, an embodiment of the present invention provides a PCIe device remote control method, in the memory read and write access phase from the remote server to the local server, the method includes:

[0108] Step S1, the PCIe hardware device sends a PCIe read / write request to the PCIe hardware device proxy component, and the PCIe hardware device proxy component receives the PCIe read / write request for the first address A; if it is a write request, the PCIe hardware device proxy component needs to synchronously cache the data to be written;

[0109] Step S2, the PCIe hardware device proxy component determines the data transmission mode between the PCIe hardware device control component and the PCIe hardware device control component according to the amount of data to be read and written: the forwarding mode of the PCIe hardware device control component and the RDMA network card direct transmission mode.

[0110] The PCIe hardware device proxy component allocates the second address interval B in the Trans transmission segment for this transmission according to the amount of data to be read and written.

[0111] Step S3, the PCIe hardware device proxy component generates a data transmission request data frame, places the data transmission request data frame in the CMD command address segment of the PCIe hardware device proxy component, generates an RDMA transmission request with the address in the CMD command address segment in the virtual PCIe device corresponding to the PCIe hardware device control component as the other end address, and transmits the generated data transmission request data frame to the PCIe hardware device control component;

[0112] The data transmission request data frame includes the address to be read or written, the data read or write request type, the read or write data length, the selected data transmission mode, and the second address interval.

[0113] Step S4, the PCIe hardware device control component receives the transmission request data frame written by the local RDMA network card to the CMD command address segment of its virtual PCIe device, and the PCIe hardware device control component parses the transmission request data frame to determine the local destination address to be accessed, i.e., the first address, the read / write request type, the data length, the transmission mode, and the second address interval B allocated by the PCIe hardware device control component;

[0114] Step S5, the PCIe hardware device control component determines a virtual third address required for RDMA transmission corresponding to the first address, and the third address is established in different ways according to different transmission modes selected by the remote PCIe hardware device proxy component;

[0115] In the forwarding mode of the PCIe hardware device control component, the PCIe hardware device control component allocates a physical fourth address D consistent with the length of the data to be transmitted in the Trans data transmission address segment space of the virtual PCIe device, and determines the virtual address corresponding to the physical fourth address D according to the memory page table established for the address segment of the virtual PCIe device during the initialization process, which is the third address C;

[0116] In the RDMA network card direct transmission mode, the PCIe hardware device control component creates a virtual address memory page table for the first address A, and the virtual address corresponding to the virtual address memory page table is the third address C, and the third address C interval is registered with the local RDMA network card;

[0117] Step S6, the PCIe hardware device control component constructs an RDMA request based on the second address B, the third address C, and the read-write access type of the PCIe hardware device proxy component, and submits the RDMA request to the local RDMA network card to trigger the RDMA network card to perform data transmission;

[0118] Step S7, when the remote PCIe hardware device proxy component receives the PCIe read / write request of the remote RDMA network card for the second address B, it performs corresponding processing according to different read / write types;

[0119] When the remote RDMA network card makes a read request to the second address B, it means that the physical PCIe device makes a write data request to the first address A. The remote PCIe hardware device proxy component returns a PCIe read response message to the RDMA network card. The data in the read response message is the data to be written by the previously cached physical PCIe device.

[0120] When the remote RDMA network card makes a write request to the second address B, it means that the physical PCIe device makes a read data request to the first address A. The remote PCIe hardware device proxy component receives the PCIe write message data of the remote RDMA network card, and encapsulates the data into a PCIe read response message, and returns it to the physical PCIe device as the read response message of the previous physical PCIe device to the read request of the first address A.

[0121] In the forwarding mode of the PCIe hardware device control component, the local RDMA network card will send a PCIe read / write request to the physical fourth address D of the PCIe hardware device control component, that is, the virtual third address C, and the PCIe hardware device control component will receive the PCIe read / write request message; if the PCIe hardware device control component receives a PCIe write request from the local RDMA network card for the fourth address D interval, the PCIe hardware device control component will cache the data to be written, determine that the final data is written to the first address A, encapsulate the cached data into a PCIe write request message, and send a PCIe write request to the first address A;

[0122] If the PCIe hardware device control component receives a PCIe read request from the local RDMA network card for the fourth address D interval, the PCIe hardware device control component determines the first address A that needs to be read and accessed, and sends a PCIe read request message to the first address A. After receiving the returned PCIe read response message, the PCIe hardware device control component extracts the data in the response message, encapsulates it into a PCIe read response message to the local RDMA network card, and returns it to the local RDMA network card.

[0123] In the RDMA network card direct transmission mode, the local RDMA network card will directly read and write the local address A, that is, the virtual third address C, and the PCIe hardware device control component does not need to intervene in the reading and writing process;

[0124] Step S8, after the data transmission is completed, the PCIe hardware device control component assists in resource recovery;

[0125] In the PCIe hardware device control component forwarding mode, the PCIe hardware device control component reclaims the allocated fourth address D interval;

[0126] In the RDMA network card direct transmission mode, the PCIe hardware device control component cooperates with the software to cancel the virtual page table mapping of address A;

[0127] Step S9, after the remote PCIe hardware device proxy component detects that the remote RDMA network card's reading and writing of the second address B interval has completely covered the current data reading and writing range, it can be determined that the current reading and writing has been completed, and the remote PCIe hardware device proxy component can reclaim the allocated second address B interval.

[0128] Example 1

[0129] The main allocation process for adding a virtual PCIe device to the local server is as follows:

[0130] Before allocation and use, the remote PCIe device management module (DEV management Remote) detects and identifies the PCIe hardware devices in its remote server and the available PCIe hardware devices in the entire cluster.

[0131] When the local server needs to add a virtual PCIe device, the local server applies for a new PCIe device through the local PCIe device management module (DEV management Local). The remote server that actually provides the physical PCIe hardware device is determined based on the number and availability of the PCIe hardware devices of each remote server, and the remote PCIe device management module of the remote server is notified. The remote PCIe device management module of the remote server allocates and determines the first PCIe hardware device (PCIe DEV1) that provides the service, as well as the matching RDMA network card. The remote PCIe device management module configures the remote PCIe hardware device agent component (DEV Agent) and the remote RDMA network card, and synchronizes the relevant information to the local PCIe device management module (DEV management Local).

[0132] The configuration of the remote DEV Agent and the remote RDMA network card by the above-mentioned remote PCIe device management module mainly includes: switching the remote DEV Agent to the remote service mode. In the remote service mode, the DEV Agent enables the space segment in its address space that serves the remote PCIe DEV (including the BAR address segment, the CFG configuration address segment, the CMD command address segment, and the Trans transmission address segment) and forwards all messages sent by the remote PCIe hardware device to the local server; the DEV Agent reads the PCIe BAR space interval of the remote PCIe DEV1, and allocates the same size and number of address spaces as the BAR interval of PCIe DEV1 in the BAR address segment of the DEV Agent; the remote PCIe device management module establishes a virtual address memory page table for the address interval in the DEV Agent that serves PCIe DEV1, and registers the Memory Region memory with the remote RDMA network card; the remote PCIe device management module creates the content required for the remote RDMA network card to establish a link, including the RDMA QP, GID, and rKey of the DEV Agent address interval;

[0133] The local PCIe device management module receives the address interval related to the PCIeDEV1 device of the DEV Agent synchronized by the remote PCIe device management module (the BAR address segment also includes the subdivided BAR space requirement information of the remote PCIe DEV1), as well as the QP, GID, and rKey information related to the RDMA network link establishment, and then allocates the address segment space Virt DEV1 space required for virtual presentation of the remote PCIe DEV1 in the local DEV Master, and records and associates the address of the corresponding address segment space in the remote DEV Agent. The local PCIe device management module establishes a virtual address memory page table for the Virt DEV1 space of the DEV Master, and registers the Memory Region memory with the local RDMA network card; the PCIe device management module allocates the local RDMA network card, and creates the content required for the local RDMA network card to establish a link, including the RDMA QP, GID, and rKey of the local Virt DEV1 address interval, and then synchronizes the address information of the local Virt DEV1 space and the local RDMA link establishment related information to the remote PCIe device management module. Then the PCIe device management module controls the local RDMA to try to establish a connection with the remote RDMA.

[0134] The remote PCIe device management module receives the Virt DEV1 space information and local RDMA link establishment information of the local DEV Master, updates the BAR start address of the remote PCIe DEV1 device to the corresponding start physical address of the BAR address segment of Virt DEV1 in the local DEV Master, associates the corresponding relationship between the address segment in the DEV Agent and the corresponding address segment space in the DEV Master, and then tries to establish a connection with the local RDMA network card.

[0135] The local RDMA network card completes the connection establishment with the remote RDMA network card.

[0136] The local PCIe device management module adds a virtual sub-PCIe bus to the virtual PCIe bus (Virt PCIe), triggering the local operating system to scan and load the Virt DEV1 device. Under the sub-PCIe bus is the device Virt DEV1, and the PCIe configuration access operation of the sub-PCIe bus is implemented by accessing the remote PCIe DEV1 through the local DEV Master. After the operating system scans and loads Virt DEV1, the PCIe BAR space address of the established Virt DEV1 device is the BAR interval address in the Virt DEV1 address segment in the local DEVMaster.

[0137] Example 2

[0138] The CPU accesses the PCIe configuration of the remote PCIe hardware device as follows:

[0139] When the CPU software needs to perform PCIe configuration access to the Virt PCIe DEV device (that is, the remote PCIe DEV device), the CPU software calls the PCIe configuration access interface of the virtual PCIe bus where the Virt PCIe DEV is located, and reads and writes to the corresponding address offset in the CPU CFG interval in the CFG configuration address segment in the DEV Master.

[0140] After receiving the PCIe read / write request from the CPU to the address in the CPU CFG area of ​​a virtual device, the DEV Master uses the corresponding address in its RDMA CFG area as the local address for RDMA network card reading and writing. If the CPU issues a write request, the data to be written by the CPU is synchronously mapped to the address corresponding to the RDMA CFG area. Then the local PCIe device management module is triggered to create an RDMA read / write request. The request type is the same as the CPU configuration read / write request type, the remote address is the corresponding address in the DEV Agent's CFG configuration space, and the length is the request length of the local CPU configuration read / write.

[0141] After the remote DEV Agent receives the PCIe read / write request from the remote RDMA network card for a certain address in the CFG configuration space of its DEV device area, it obtains the address that needs to be accessed for the device configuration according to the read / write address mapping, and then generates a PCIe configuration read / write request for the remote physical PCIe DEV according to the read / write type, read / write configuration space address, read / write length, and if it is a write request, it also contains the data to be written, and sends it to the PCIe DEV. After the remote PCIe device management module receives the configuration read / write response information returned by the PCIe DEV, it processes it in different ways according to the read / write type: if it is a read request, the data in the configuration read response returned by the PCIe DEV is extracted and used as the response data of the remote RDMA read request to respond to the remote RDMA read request; if it is a write request, the remote PCIe device management module is triggered to create an RDMA request for configuration write to return a configuration write response to the local DEV Master. The addresses at both ends of the created RDMA request are the pre-agreed addresses in the CMD command address area, and the content includes the status information of the write completion.

[0142] When the local DEV Master receives configuration read data in the RDMA CFG configuration area or write response information in the CMD address area, it is ready to respond to the CPU's configuration read and write requests. For CPU configuration read requests, it returns a PCIe read response message to the CPU; for CPU configuration write requests, it notifies the CPU's configuration write interface function software through an interrupt or write flag bit to complete the configuration write.

[0143] Example 3

[0144] When the CPU needs to read and write the BAR address of the virtual device, since the BAR address of the virtual device is configured as the BAR address segment interval of the virtual device in the local DEV Master during device scanning, the CPU actually directly performs PCIe read and write access to the BAR address segment of the virtual device of the DEVMaster.

[0145] When the DEV Master receives a PCIe read or write request for a BAR address A in a virtual device area, if it is a write request, it will synchronously cache the data to be written, allocate an address space segment B equal to the size of the data request in the Trans segment of the virtual device area, and establish an associated mapping with the current BAR read or write request address interval. Then, it will trigger the local PCIe device management module software to create an RDMA request. The requested address is address B allocated in the Trans area of ​​the local DEV Master and address C corresponding to address A in the device BAR of the remote DEV Agent. The read or write length is the requested data length, and the type is the read or write type requested by the CPU.

[0146] When DEV Master receives a read or write request from the local RDMA network card for address B, if it is a write request, it caches the data and uses it as the response message data of the CPU's read request for address A, and returns a read response to address A to the CPU; if it is a read request, it uses the previously cached CPU write data for address A as the RDMA read response message data, and returns a read response to RDMA.

[0147] After receiving the read and write request from the remote RDMA network card to the pool C, the remote DEV Agent's DEV's BAR address segment determines the remote physical PCIe DEV device's pending address D according to the mapping relationship configured during initialization, and sends a read and write request to the PCIe DEV device's address D. If it is a read request, it waits for the PCIe DEV's read request response, and then responds to the RDMA read request with the read request response data as the data of the RDMA read request response message.

[0148] After completing returning the read response message to the CPU or completing returning the read response message to the local RDMA network card, the DEV Master reclaims the address interval B for auxiliary data transmission allocated in its Trans area.

[0149] Example 4

[0150] The fused CPU reads and writes memory to the remote PCIe DEV:

[0151] Since ordinary PCIe read and write messages can be transmitted asynchronously, the DEV Master can receive multiple read or write requests and then respond asynchronously. The DEV Master can perform fusion processing on multiple read or write requests.

[0152] After receiving a CPU read or write request for the virtual device BAR address segment address, DEV Master caches the request and waits for other subsequent read or write requests to arrive. DEV Master processes read and write requests separately and classifies the requests by read and write type. After exceeding the predetermined time or the amount of data to be processed reaches the predetermined value, DEV Master uniformly allocates addresses in the local Trans address segment based on the aggregated read or write requests, and triggers the local PCIe device management module software to create RDMA requests, and makes unified data read and write requests to the remote DEV Agent.

[0153] Example 5

[0154] The remote PCIe DEV reads and writes requests to the local memory address:

[0155] When the remote PCIe DEV needs to read or write local memory address A, it will send a PCIe read or write request message to the remote DEV Agent. The address of the read or write request in the message is A. If it is a write request, it will also include the data to be written.

[0156] After receiving the read / write request for address A, DEV Agent allocates an address space size consistent with the data volume of this read / write request in the Trans area of ​​DEV Agent, and the address is B. If it is a write request, the data to be written is cached synchronously. DEVMaster selects the transmission mode to be used on the DEV Master side according to the amount of data to be read / written and the status of the task to be processed: DEV Master forwarding mode or RDMA network card direct transmission mode; creates a command request data frame in the predetermined address in the CMD command address segment of the device in DEV Agent, and the data frame contains the requested address A, data length, transmission mode used by DEV Master, and transmission address B in DEV Agent, and then triggers DEV Mngt to generate an RDMA request to transmit the request data frame to the predetermined address in the CMD command address segment in the virtual device of DEV Master.

[0157] After the CMD address segment of the virtual device in the DEV Master receives the request data frame sent by the DEV Agent, it performs corresponding processing according to the transmission mode set by the DEV Agent:

[0158] When using the DEV Master forwarding mode, the DEV Master allocates address C in the Trans address segment of the corresponding virtual device as the RDMA transmission address and triggers the local PCIe device management module to create an RDMA request;

[0159] When using the RDMA network card direct transmission mode, the DEV Master triggers the DEV Mngt to access the virtual address memory page table of the local address A range, and creates an RDMA request with address A (the virtual address corresponding to A).

[0160] If the DEV Master forwarding mode is used, when the DEV Master receives the PCIe read / write request message for address C from the local RDMA network card, it will send a PCIe read / write message to address A. If it is a read request, the read result data will be synchronously returned to the RDMA network card in the form of a PCIe response message. After the transmission is completed, the address interval C allocated in the Trans address segment will be reclaimed. If the RDMA network card direct transmission mode is used, the DEV Agent does not need to directly intervene in the data transmission. After the RDMA transmission is completed, the remote PCIe device management module software releases the virtual memory page table of the address interval A.

[0161] After receiving the read / write request for address B from the remote RDMA network card, if it is a read request, the DEV Agent returns the data to be written previously sent by the remote physical PCIe DEV device as a PCIe response message to the RDMA network card; if it is a write request, the data to be written in the request is returned to the remote physical PCIe DEV device in the form of a read response message.

[0162] The above mainly implements the allocation, configuration and access process of virtual PCIe devices. Through these access process implementations, common usage interactions between a typical PCIe device and application software can be realized, thereby realizing the remote use of physical PCIe devices. Note that the processes and implementation objects in the above implementation process do not need to be completely consistent with the process described above. For example, when a remote device reads and writes memory to a local device, the transmission method can also be determined by the PCIe hardware device control component (DEV Master).

[0163] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the PCIe device remote control method as described above. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability in which the PCIe device remote control method provided by the embodiment of the present invention is located, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0164] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the PCIe device remote control method as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A PCIe device remote control method, characterized in that: During the initialization phase, the method includes: Starting the local server, and pre-allocating the address space of each virtual PCIe device in the PCIe hardware device control component; When the local server accesses the remote PCIe hardware device, determine the remote server that needs to provide the remote PCIe hardware device, determine the remote PCIe hardware device and the remote PCIe hardware device to be accessed, and remove the remote PCIe hardware device to be accessed from the logical layer of the remote server; Configure the PCIe hardware device proxy component corresponding to the remote PCIe hardware device, and set the working mode of the PCIe hardware device proxy component to the remote service mode; Allocate a PCIe BAR space address range in the PCIe hardware device control component for the virtual PCIe device; Obtain the PCIe BAR space requirement corresponding to the remote PCIe hardware device, and reallocate the PCIe BAR space address corresponding to the remote PCIe hardware device according to the PCIe BAR space start address corresponding to the virtual PCIe device in the PCIe hardware device control component, so that the PCIe BAR space address of the remote PCIe hardware device is the same as the PCIe BAR space address of the virtual PCIe device in the PCIe hardware device control component; and synchronize the PCIe BAR space address of the remote PCIe hardware device to the corresponding PCIe hardware device proxy component; Establishing virtual address information for address spaces related to the PCIe hardware device in the PCIe hardware device control component and the PCIe hardware device proxy component; The address information of the virtual PCIe device in the PCIe hardware device control component is exchanged with the address information of the PCIe hardware device proxy component to obtain the complete address information; and the complete address information is registered in the memory of the local RDMA network card and the remote RDMA network card respectively to establish a connection between the local RDMA network card and the remote RDMA network card; Configure a virtual PCIe bus and add a sub-bus under the virtual PCIe root bus. The sub-bus corresponds to the virtual PCIe device in the PCIe hardware device control component.

2. A PCIe device remote control method according to claim 1, characterized in that: In the PCIe configuration read and write phase, the method comprises the following steps: Call the virtual PCIe bus configuration read and write interface to send a configuration read and write request to the corresponding address in the Config configuration space where the virtual PCIe device in the PCIe hardware device control component is located; When the PCIe hardware device control component receives a configuration read / write request, it encapsulates the configuration read / write request into an RDMA write data request, communicates with the remote RDMA network card through the local RDMA network card, and sends the RDMA write data request to the config configuration space of the PCIe hardware device proxy component; sends the RDMA read data request according to the result data address in the config configuration space of the PCIe hardware device proxy component; When the PCIe hardware device proxy component receives a configuration read / write request transmitted by a remote RDMA network card, it generates and sends a PCIe configuration read / write message to the PCIe hardware device, and the bus number in the PCIe configuration read / write message is the bus number of the physical PCIe bus where the PCIe hardware device is located; After receiving the configuration read / write completion response message returned by the PCIe hardware device, the PCIe hardware device proxy component places the configuration read / write result at the result data access address of its configuration space; When the PCIe hardware device proxy component receives a PCIe read request for configuration result data sent by the remote RDMA network card, it returns the read / write result data to the remote RDMA network card in the form of a PCIe read / write response message; When the PCIe hardware device control component receives a PCIe write data request from the local RDMA network card to the result data storage address in its config configuration space, it calls the configuration read and write interface of the virtual PCIe bus; when the configuration read and write is completed, it returns the configuration write request or returns the configuration read request together with the configuration data.

3. A PCIe device remote control method according to claim 2, characterized in that: The read / write request includes the device number, PCIe function number, configuration access register address, and read / write type; if it is a write request, it also needs to include the data information to be written.

4. A PCIe device remote control method according to claim 1, characterized in that: The destination address of the RDMA read data request is the storage address of the result data in the config configuration space in the PCIe hardware device control component.

5. A PCIe device remote control method according to claim 1, characterized in that: In the memory read and write access phase from the local server to the remote server, the method includes: When the local CPU performs memory read and write to the PCIe BAR address of the virtual PCIe device, a memory read and write request is sent to the PCIe BAR address of the corresponding virtual PCIe device in the PCIe hardware device control component; When the PCIe hardware device control component receives a PCIe memory read / write request for the first address, the PCIe hardware device proxy component receives a second address corresponding to the first address according to a mapping relationship between the local BAR address and the BAR address of the PCIe hardware device proxy component. Submit an RDMA read / write request and perform RDMA transmission through a local RDMA network card and a remote RDMA network card; the RDMA read / write request is consistent with the memory read / write request type received by the PCIe hardware device control component; the source address and destination address of the RDMA read / write request are a local first address and a remote second address respectively; When the local CPU issues a memory write request, after the local server and the remote server start the RDMA request, the local RDMA network card will send a PCIe read data request to the first address in the PCIe hardware device control component. After receiving the PCIe read data request sent by the local RDMA network card, the PCIe hardware device control component returns the cached data to be written as the data in the PCIe read completion message to the local RDMA network card, and the local RDMA network card sends the write request and the data to be written to the remote RDMA network card. After the remote RDMA network card receives the write request and the data to be written transmitted by the local RDMA network card, the remote RDMA network card sends a PCIe write request to the second address of the PCIe hardware device proxy component; after the PCIe hardware device proxy component receives the PCIe write request sent by the remote RDMA network card, it caches the data to be written, calculates the corresponding third address in the PCIe hardware device according to the second address, and sends a PCIe write request message to the PCIe hardware device; wherein the destination address of the PCIe write request message is the third address, and the data of the PCIe write request message is the cached data to be written; or, When the local CPU issues a memory read request, after the local server and the remote server start the RDMA request, the local RDMA network card receives the read data returned by the PCIe hardware device proxy component sent by the remote RDMA network card, and the local RDMA network card sends a PCIe write data request to the first address in the PCIe hardware device control component. After receiving the PCIe write data request, the PCIe hardware device control component caches the write request data and returns the write request data to the local CPU as the data in the completion message of the local CPU's PCIe read data request for the first address; the local RDMA network card sends a read request to the remote RDMA network card; The remote RDMA network card will receive the read request transmitted by the local RDMA network card, and send a PCIe read request to the second address of the PCIe hardware device proxy component; after receiving the PCIe read request sent by the remote RDMA network card, the PCIe hardware device proxy component calculates the corresponding third address in the PCIe hardware device according to the second address, and sends a PCIe read request message to the PCIe hardware device; wherein the destination address of the PCIe read request message is the third address; after receiving the PCIe read response message returned by the PCIe hardware device, the PCIe hardware device proxy component extracts data in the PCIe read response message, and returns the PCIe read response message and data to the read request of the remote RDMA network card.

6. A PCIe device remote control method according to claim 1, characterized in that: In the memory read and write access phase from the remote server to the local server, the method includes: The PCIe hardware device sends a PCIe read / write request to the PCIe hardware device proxy component, and the PCIe hardware device proxy component receives the PCIe read / write request for the first address; The PCIe hardware device proxy component determines the data transmission mode between the PCIe hardware device control component and the second address interval in the Trans transmission field used for this transmission according to the amount of data to be read and written; The PCIe hardware device proxy component generates a data transmission request data frame, places the data transmission request data frame in the CMD command address segment of the PCIe hardware device proxy component, generates an RDMA transmission request with the address in the CMD command address segment in the corresponding virtual PCIe device in the PCIe hardware device control component as the other end address, and transmits the generated data transmission request data frame to the PCIe hardware device control component; The data transmission request data frame includes the address to be read or written, the data read or write request type, the read or write data length, the selected data transmission mode, and the second address interval; The PCIe hardware device control component receives a transmission request data frame written by the local RDMA network card to the CMD command address segment of its virtual PCIe device, and the PCIe hardware device control component parses the transmission request data frame to determine the local destination address to be accessed, i.e., the first address, the read / write request type, the data length, the transmission mode, and the second address allocated by the PCIe hardware device control component; The PCIe hardware device control component determines a virtual third address required for RDMA transmission corresponding to the first address; The PCIe hardware device control component constructs an RDMA request based on the second address, the third address, and the read / write access type of the PCIe hardware device proxy component, and submits the RDMA request to the local RDMA network card to trigger the RDMA network card to perform data transmission; When the remote PCIe hardware device proxy component receives a PCIe read / write request for the second address from the remote RDMA network card, it includes: When the remote RDMA network card makes a read request to the second address, that is, the physical PCIe device makes a write data request to the first address, the remote PCIe hardware device proxy component returns a PCIe read response message to the RDMA network card, and the data in the read response message is the data to be written by the previously cached physical PCIe device; When the remote RDMA network card makes a write request to the second address, it means that the physical PCIe device makes a read data request to the first address. The remote PCIe hardware device proxy component receives the PCIe write message data of the remote RDMA network card, and encapsulates the data into a PCIe read response message, and returns it to the physical PCIe device as a read response message of the previous physical PCIe device to the read request of the first address; After the remote PCIe hardware device proxy component detects that the reading and writing of the second address interval by the remote RDMA network card has completely covered the current data reading and writing range, the current reading and writing is completed.

7. A PCIe device remote control method according to claim 6, characterized in that: The data transmission modes include: forwarding mode of PCIe hardware device control component and direct transmission mode of RDMA network card; The process in which the PCIe hardware device control component determines the virtual third address required for RDMA transmission corresponding to the first address includes: In the forwarding mode of the PCIe hardware device control component, the PCIe hardware device control component allocates a physical fourth address consistent with the length of the data to be transmitted in the Trans data transmission address segment space of the virtual PCIe device, and determines the virtual address corresponding to the physical fourth address according to the memory page table established for the address segment of the virtual PCIe device during the initialization process, which is the third address; In the RDMA network card direct transmission mode, the PCIe hardware device control component creates a virtual address memory page table for the first address, the virtual address corresponding to the virtual address memory page table is the third address, and the third address interval is registered with the local RDMA network card; The process in which the remote PCIe hardware device proxy component receives the PCIe read / write request for the second address from the remote RDMA network card includes: In the forwarding mode of the PCIe hardware device control component, the local RDMA network card will send a PCIe read / write request to the physical fourth address of the PCIe hardware device control component, that is, the virtual third address, and the PCIe hardware device control component will receive the PCIe read / write request message; if the PCIe hardware device control component receives a PCIe write request from the local RDMA network card to the fourth address interval, the PCIe hardware device control component will cache the data to be written, determine that the final data is written to the first address, encapsulate the cached data into a PCIe write request message, and send a PCIe write request to the first address; If the PCIe hardware device control component receives a PCIe read request for the fourth address interval from the local RDMA network card, the PCIe hardware device control component determines the first address that needs to be read and accessed, and sends a PCIe read request message to the first address. After receiving the returned PCIe read response message, the PCIe hardware device control component extracts the data in the response message, encapsulates it into a PCIe read response message for the local RDMA network card, and returns it to the local RDMA network card. In the RDMA network card direct transmission mode, the local RDMA network card will directly read and write the local address, that is, the virtual third address, and the PCIe hardware device control component does not need to intervene in the reading and writing process.

8. A PCIe device remote control device, characterized in that: Used to implement the PCIe device remote control method as described in any one of claims 1-7, the device comprises: a local server and a remote server; Wherein, the local server is deployed with a PCIe hardware device control component, a virtual PCIe bus, and several local RDMA network cards; The remote server is deployed with a PCIe hardware device, a PCIe hardware device proxy component, and several remote RDMA network cards; The PCIe hardware device control component is used to realize the virtual PCIe device presentation of the PCIe hardware device deployed on the remote server on the local CPU; the virtual PCIe bus is used to manage the virtual PCIe bus and the virtual PCIe devices on the bus for the local CPU; The PCIe hardware device proxy component is paired one-to-one with the PCIe hardware device deployed on the remote server, and is used to complete the proxy forwarding of the PCIe request between the remote RDMA network card and the PCIe hardware device.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the PCIe device remote control method described in any one of claims 1-7 above.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the PCIe device remote control method as described in any one of claims 1 to 7 is implemented.

Images