Heterogeneous computing system equipment sharing device and method based on PCIe Switch

Abstract

The present invention relates to a device and method for sharing heterogeneous computing systems based on PCIe Switch, which allows devices in an interconnected system to be shared in all nodes, wherein a borrowing module is used to obtain the PCIe device status of a lender, obtain the PCIe device configuration information of the lender when the PCIe device of the lender can be borrowed, create a virtual PCIe device based on the acquired configuration information, access the PCIe device of the lender through the virtual PCIe device, and return and release the PCIe device resources of the lender after exiting the access; the lending module is used to notify the borrower of the local PCIe device status, and reclaim the lent PCIe device resources after the borrower exits the access to the PCIe device; the node event dynamic monitoring module is used to monitor the node status in the interconnected system, establish a handshake connection with the newly added node, determine whether there is a connection relationship with the exited node, and perform processing including exiting the connection or reclaiming the lent PCIe device resources according to the judgment result.

Description

Technical Field

[0001] The present invention relates to the technical field of PCIe Switch, and in particular to a heterogeneous computing system equipment sharing device and method based on PCIe Switch. Background Art

[0002] So far, after years of updates and iterations, the performance of chips has been continuously improved, but due to constraints such as production and manufacturing, the improvement of chip performance has reached a bottleneck, and it is difficult to have a major breakthrough in a short period of time. And with the explosive development of the Internet and the rise of artificial intelligence in recent years, traditional CPU processors are difficult to meet the growing demand for computing power, and problems such as insufficient bandwidth, high latency, and high overhead have emerged. Therefore, heterogeneous computing networks equipped with acceleration chips such as GPU / NPU / FPGA / DPU and building CPU+XPU have gradually emerged. Compared with a single computing architecture, heterogeneous computing uses multi-node interconnection and communication technology to achieve horizontal collaboration in the three dimensions of computing, networking, and storage between different nodes, thereby improving computing performance and reducing power consumption costs.

[0003] The pooling and sharing of device resources of each node in a heterogeneous computing system has become a primary issue. Sharing resources between nodes and using them on demand can greatly reduce the cost of computing power, which requires that the central processor of each node not only has the ability to control its own node devices, but also has the ability to control the remote host node devices. For example, when a machine needs more device resources, additional devices can be dynamically borrowed from other machines with redundant devices. This method allows device resources to be shared among multiple computers, increasing the flexibility and resource utilization of the system. This means that we can use more cost-effective hardware devices instead of blindly using more powerful and more expensive devices, thereby greatly reducing the cost of computing power.

[0004] The invention patent with application number 202410031395 discloses a vehicle resource sharing method, apparatus, device, medium and vehicle based on PCIe. An initial PCIe device is created in the first controller in the vehicle. The first controller obtains the configuration information of the shared PCIe device in the second controller through NTB. The shared PCIe device is a PCIe device that the first controller and the second controller need to share. The first controller configures the initial PCIe device based on the configuration information to obtain the target PCIe device. The first controller accesses the shared PCIe device through the target PCIe device. According to this embodiment, PCIe device sharing between the first controller and the second controller can be achieved, thereby reducing the cost of resource configuration and the cost of wiring. Moreover, when the first controller accesses the shared PCIe device in the second controller, the access method is close to local access, and no protocol conversion and parsing are required, thereby effectively reducing the delay of data transmission. However, this prior art still has many shortcomings, as described below:

[0005] (1) Lack of dynamic and flexible adjustment capabilities for device resources: In this system, the first controller and the second controller share devices through NTB, but there is no mention of device recycling and redistribution. In actual usage scenarios, the interconnected system may be very large, and devices need to be shared among all nodes, which requires that the devices can be correctly recycled when needed and redistributed according to the current computing power requirements.

[0006] (2) Lack of dynamic host node joining and exiting capabilities: In this system, the first controller and the second controller share devices through NTB, but if one of the controllers fails, the entire system will be paralyzed, which is fatal in the vehicle system. Therefore, the device borrowing solution in the interconnected system needs to support the dynamic exit of the host node and ensure the normal operation of other nodes in the system. In addition, when other nodes join the system, the device resources of the node should also be able to flexibly join the resource pool.

[0007] (3) Single usage scenario: The system is applied in the vehicle field and the usage scenario is relatively single. Summary of the invention

[0008] When the inventors of this application conducted a study on the resource pooling and device borrowing method of a heterogeneous computing system based on PCIe Switch, they found that the prior art was not mature, lacked the ability to dynamically and flexibly adjust device resources, and did not consider the dynamic borrowing of device resources between various nodes. When the inventors of this application conducted a survey on the device borrowing scheme of a heterogeneous computing system, they found that the heterogeneous computing interconnection system was very complex, and the allocation of computing power might change in real time. For example, a certain node needs a large amount of storage resources at certain times, so the node needs to borrow storage devices in the system resource pool, but after a period of time, the node is idle, and the storage device resources originally occupied need to be returned to the resource pool for use by other nodes. In addition, the inventors of this application also found during the survey that a complex interconnection system must have the ability to dynamically join and exit. When a single node fails, the failure should not affect the normal operation of other nodes in the system. At the same time, the device resources occupied by the failed node should be correctly recovered for use by other computing nodes in the system. When a new computing node joins, the device resources of the node can also be flexibly added to the device resource pool.

[0009] The purpose of the present invention is to provide a PCIe Switch-based heterogeneous computing system resource pooling and device borrowing method to solve the following problems in the above-mentioned technology: (1) lack of dynamic and flexible adjustment capability of device resources; (2) lack of dynamic node joining and exiting capability; (3) single usage scenario problem.

[0010] The heterogeneous computing system device sharing device based on PCIe Switch proposed by the present invention is used to share PCIe devices in an interconnected system among all nodes, and the nodes are divided into borrowers and lenders according to the borrowing relationship, and are characterized in that the nodes have a borrowing module, a lending module and a node event dynamic monitoring module;

[0011] The borrowing module is used to manage the PCIe device belonging to the lender, including obtaining the PCIe device status of the lender, obtaining the PCIe device configuration information of the lender when the PCIe device belonging to the lender can be borrowed, creating a virtual PCIe device based on the obtained configuration information, accessing the PCIe device belonging to the lender through the virtual PCIe device, and returning and releasing the PCIe device resources belonging to the lender after exiting the access;

[0012] The lending module is used to manage the local PCIe device, including notifying the borrower of the local PCIe device status and reclaiming the lent PCIe device resources after the borrower quits accessing the PCIe device;

[0013] The node event dynamic monitoring module is used to monitor the node status in the interconnected system, establish a handshake connection with the newly added node, determine whether there is a connection relationship with the exiting node, and perform processing including exiting the connection or recovering the loaned PCIe device resources based on the judgment result.

[0014] As a further improvement, in the event of a node joining, the local node sends a handshake message to the newly joined node through the newly joined node ID fed back by the underlying layer. After the newly joined node receives the handshake message, it returns a reply message in the message callback function to form a handshake connection between the newly joined node and the local node.

[0015] As a further improvement, in the event of a node exit, the local node first traverses all instance instances and determines whether the node has a connection relationship with the faulty node based on the instance id. If there is no corresponding instance id, it means that the connection no longer exists. In this case, it can exit directly. When there is a connection relationship, resource recycling is performed.

[0016] The present invention also proposes a heterogeneous computing system device sharing method based on PCIe Switch, comprising the following steps:

[0017] The lender and the borrower send messages to each other, establish a handshake connection, and create an instance of the other end;

[0018] The lender adds the sharable PCIe device to the device pool and sets the status of the sharable PCIe device to be usable;

[0019] The lender actively connects to the borrower and creates a shared memory area so that both parties can exchange PCIe device status;

[0020] The lender broadcasts the PCIe device status to notify the borrower;

[0021] The borrower initiates a borrowing request for a PCIe device, and the lender returns the configuration information of the PCIe device;

[0022] The borrower creates a virtual PCIe device based on the configuration information returned by the lender, and the borrower's access to the virtual PCIe device is transparently forwarded to the lender's PCIe device;

[0023] After the borrower obtains the control right of the PCIe device belonging to the lender, the lender marks the status of the PCIe device as being in use and prevents other nodes in the interconnected system from accessing the PCIe device;

[0024] After the borrower quits accessing the PCIe device belonging to the lender, the lender marks the status of the PCIe device as usable, and other hosts in the interconnected system can borrow the PCIe device.

[0025] As a further improvement, when the borrower initiates a borrowing request for a PCIe device, the borrower actively intercepts the configuration read and write methods of the PCIe device, and transfers the configuration read and write requests of the PCIe device to the lender through RPC remote calls, and the lender accesses the configuration space of the PCIe device.

[0026] A further improvement is that when the borrower initiates a borrowing request for a PCIe device, the borrower determines the BDF of the PCIe device through the hook function. If the device being accessed is a remote device, an RPC remote call is initiated to complete the reading and writing of the actual configuration of the PCIe device. At the same time, the borrower maintains the configuration space of the PCIe device in the local memory buffer and reads directly from the local cache when a remote call is not required.

[0027] As a further improvement, the configuration information includes configuration space parameters and memory address space parameters.

[0028] As a further improvement, the configuration information of the virtual PCIe device is identical to the configuration information of the shared PCIe device.

[0029] As a further improvement, the configuration read request interception process of the shared PCIe device includes the following steps:

[0030] The borrower host issues a read request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration read method is used. If it is a shared device, the borrower initiates an RPC remote call to notify the lender host to complete the read operation of the device configuration space. The lender host reads the data and returns the result to the borrower host. The borrower host saves the return value in the local cache.

[0031] As a further improvement, the configuration write request interception process of the shared PCIe device includes the following steps:

[0032] The borrower host issues a write request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration write method is used. If it is a shared device, the data is first written to the local cache, and then the borrower initiates an RPC remote call to notify the lender host to complete the write operation of the device configuration space. The lender host receives the request and writes the data to the device's configuration space.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] Compared with the prior art, the device borrowing software solution of the present application allows the devices in the interconnected system to be shared among all nodes without the need for any software middle layer. The device driver runs on a machine physically separated from the device, and can directly control the remote device as if the device and the driver are on the same machine, without modifying the driver, truly achieving software-unawareness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart for configuring read request interception.

[0036] Figure 2 Write a request interception flow chart for configuration.

[0037] Figure 3 Borrow flow chart for PCIe devices.

[0038] Figure 4 Inject flow graph for PCIe devices. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0040] Definitions of Abbreviations and Key Terms

[0041] PCIe: Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard, mainly used to expand computer system bus data throughput and improve device communication speed;

[0042] PCIe Switch: PCI Express (PCIe) switch, a device that allows multiple PCIe devices to connect to the same PCIe controller or PCIe host;

[0043] ID: Identification, usually refers to a number or code used to uniquely identify someone or something;

[0044] NTB: Non-Transparent Bridging, a method of solving the address conflict and packet routing problems between systems through address conversion and packet translation;

[0045] BDF: A triplet that uniquely identifies a device on the PCIe bus;

[0046] CPU: Central Processing Unit, is the computing and control core of the computer system, and is the final execution unit for information processing and program running;

[0047] GPU: Graphics Processing Unit, a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.);

[0048] NPU: Neural Network Processing Unit, a processor designed specifically for processing data for neural networks and machine learning;

[0049] FPGA: Field Programmable Gate Array, refers to field programmable gate array;

[0050] DPU: Data Processing Unit, refers to data processor;

[0051] BARs: Base Address Registers, a set of registers used to describe the resources required by PCIe devices, such as memory address space and I / O port space;

[0052] MSI: Message Signaled Interrupts, the PCIe device triggers a CPU interrupt by writing a specific message to a specific address;

[0053] DMA, Direct Memory Access, is used for data transfer;

[0054] RPC: Remote Procedure Call, remote procedure call;

[0055] DEVFN: Device Function ID, the identifier of the PCIe device;

[0056] I / O port: Input / Output Port, is the channel used for data exchange with external devices in the computer system, and is the interface between computer hardware and external devices;

[0057] 0xCF8: starting address of PCI configuration space address port;

[0058] 0xCFC: Another I / O port address accessed by the PCI configuration space, used in conjunction with 0xCF8 to form the PCI configuration access mechanism;

[0059] CFG_ADDR: A special I / O port address used to access the configuration space of the PCI device;

[0060] CFG_DATA: Another I / O port address related to PCI device configuration space access, used in conjunction with the CFG_ADDR (0xCF8) port to form the PCI configuration access mechanism;

[0061] X86: A widely used microprocessor architecture;

[0062] ARM: Acorn RISC Machine, a widely used 32-bit and 64-bit RISC microprocessor architecture;

[0063] PowerPC: Performance Optimization With Enhanced RISC - PerformanceComputing, a 32-bit and 64-bit microprocessor architecture based on the RISC architecture;

[0064] MMIO: Memory-Mapped I / O, a common input / output (I / O) interface mechanism that maps the registers of peripheral devices to the computer's memory address space and accesses these device registers in the same way as accessing memory;

[0065] Memory address: It is the address used to uniquely identify the location of a storage unit in a computer system;

[0066] Linux: an open source operating system kernel;

[0067] EVT_HOT_ADD: A system event indicating that a new hardware device has been added during system operation;

[0068] EVT_HOT_REMOVE: A system event indicating that a hardware device has been removed or hot-plugged while the system is running;

[0069] IOMMU: Input-Output Memory Management Unit. IOMMU is a hardware unit located between the device and the main memory. It is responsible for managing the device's direct access to the memory. It prevents the device from accessing the memory area across boundaries through address translation and memory protection mechanisms, ensuring data security and system stability.

[0070] Instance: It is the specific individual or concretization of an object.

[0071] At present, each node in the host cluster has a demand for the use of PCIe devices, which requires the configuration of corresponding PCIe devices for each node in the host cluster, and thus requires a large number of PCIe devices, which is costly. In view of this, an embodiment of the present application provides a device sharing method for a heterogeneous computing system based on PCIe Switch, which can allow any node in the host cluster to control the PCIe devices of other nodes. For the convenience of distinction and description, the node that needs to share the PCIe device belonging to other nodes is called a borrower, and the node to which the shared PCIe device belongs is called a lender. The shared PCIe device refers to a real PCIe device, which is plugged into the node for shared use by other nodes in the host cluster.

[0072] In this embodiment, the borrower and the lender send handshake information to each other to establish a handshake connection. The borrower and the lender can exchange their own PCIe device status information, which is stored in the shared memory area created by the borrower and the lender.

[0073] When it is necessary to borrow PCIe devices from other nodes, the borrower can read the status information of the PCIe devices in the shared memory area, and after determining the shared PCIe device to be accessed, send a configuration space read / write request to the lender to which the shared PCIe device belongs. The configuration space read / write request is used to request the lender to send the configuration information of the shared PCIe device to the borrower. After receiving the configuration space read / write request, the lender sends the configuration information of the shared PCIe device to the borrower in response to the configuration space read / write request. In this way, the borrower can obtain the configuration information of the shared PCIe device.

[0074] The configuration information content of the shared PCIe device includes the basic information of the shared PCIe device, including but not limited to configuration space parameters, memory address space parameters, etc. Among them, the configuration space parameters refer to the parameters of the configuration space, and the configuration space parameters include but are not limited to the device-specific information, device status information, interrupt information, etc. contained in the configuration space. The memory address space parameters refer to the parameters of the memory address space, that is, the parameters of the memory address space, and the memory address space parameters include the memory addresses allocated to each function in the PCIe device, for example, they may include: BAR memory address, MSI memory address, DMA memory address, etc.

[0075] Since the shared PCIe device is a real device, that is, the content of the configuration space of the shared PCIe device is known content, therefore, the content of the configuration space of the shared PCIe device can be obtained.

[0076] Under normal circumstances, a computer system only supports one PCIe address domain, that is, different computer systems have their own independent PCIe address domains, resulting in the inability to directly communicate data between different PCIe address domains. For example, in an X86 processor, the contents of the PCIe device configuration space are accessed through the two I / O ports 0xCF8 and 0xCFC. In a PowerPC processor, the contents of the PCIe device configuration space are accessed through the CFG_ADDR and CFG_DATA registers. In an ARM processor, the contents of the PCIe device configuration space are accessed through the MMIO method. In order to abstract these differences, the Linux kernel's access to the configuration space is completed using a unified interface function. In this embodiment, the configuration space read and write request issued by the borrower cannot penetrate through the address conversion of the NTB to reach the configuration space of the remote shared PCIe device, so the borrower can achieve data communication with the lender through the RPC remote system component.

[0077] After receiving the configuration information of the shared PCIe device, the lender can create a virtual PCIe device locally and inject the virtual PCIe device into the local PCI tree. The virtual PCIe device is not a real PCIe device, but a virtualized virtual PCIe device, and the configuration information of the virtual PCIe device is consistent with the configuration information of the shared PCIe device.

[0078] When the borrower accesses the virtual PCIe device, the access is transparently forwarded to the shared device of the lender. This way, the borrower's access to the remote shared device is no different from local access, and can produce the same impact. To achieve this goal, the device creation process can be replicated in the kernel. However, the device scanning and management code in the kernel is not simple, and copying and understanding all the code involved requires a lot of work. In addition, after the virtual PCIe device is created, the virtual PCIe device still needs to be handed over to the kernel. Since the PCI subsystem may decide to modify or release the memory associated with the device, the memory management of the device will become difficult. Another simpler method is to make the PCI subsystem think that it sees a new device and initialize it itself. The Linux kernel uses configuration read and write to obtain information about the device. By intercepting access to the configuration space, the kernel can be deceived to see a virtual device and let the kernel initialize the device structure itself, so that the kernel code does not need to be modified at all, thereby minimizing the impact on the native system. In this embodiment, by intercepting the read and write access to the remote device configuration, the remote shared PCIe device is injected into the same bus as the NTB.

[0079] like Figure 4 As shown, the PCIe device injection process includes the following steps:

[0080] First, get the PCIe device descriptor according to the device BDF number and determine whether the descriptor is empty. If it is not empty, it means that the same device has been injected before and cannot be injected repeatedly. The injection fails and returns. If the descriptor is not empty, initialize the device and attach the device to the PCI subsystem. Then allocate various resources to the device and bind the device to the IOMMU group to which the NTB belongs. Then replace the device BAR space address with the window address in the NTB, then install the DMA hook function for the device, and finally bind the device driver to complete the device injection process.

[0081] When the borrower node accesses the configuration space of the PCIe device, the BDF of the PCIe device can be determined by means of hooks. If the PCIe device accessed is remote, an RPC remote call is initiated to complete the real configuration reading and writing of the shared PCIe device. At the same time, the borrower node also needs to maintain the configuration space of the shared PCIe device in the local memory buffer. When a remote call is not required, it is read directly from the local cache, thereby reducing the number of RPC remote calls and reducing system overhead.

[0082] like Figure 1 As shown, the configuration read request interception process of the shared PCIe device includes the following steps:

[0083] First, the borrower host issues a read request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration read method is used; if it is a shared device, the borrower initiates an RPC remote call to notify the lender host to complete the read operation of the device configuration space. The lender host reads the data and returns the result to the borrower host, and the borrower host saves the return value in the local cache.

[0084] like Figure 2 As shown, the configuration write request interception process of the shared PCIe device includes the following steps:

[0085] First, the borrower host issues a write request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration write method is used; if it is a shared device, the data is first written to the local cache, and then the borrower initiates an RPC remote call to notify the lender host to complete the write operation of the device configuration space. The lender host receives the request and writes the data to the device configuration space.

[0086] After the borrower obtains the control right of the shared PCIe device, it notifies the lender, and the lender updates the status of the shared PCIe device to being in use, and other nodes in the host cluster cannot use the shared PCIe device;

[0087] After the borrower has finished using the shared PCIe device, the shared PCIe device is released and the lender is notified. The lender updates the status of the shared PCIe device to an available status, and other nodes in the host cluster can use the shared PCIe device.

[0088] like Figure 3 As shown in the figure, the overall process of equipment borrowing includes the following steps:

[0089] The lender and the borrower send messages to each other, establish a handshake connection, and create an instance of the other end;

[0090] The lender adds the device to the device pool, creates a shared device, and sets the device to a borrowable state;

[0091] The lender actively connects to the borrower and creates a shared memory area so that both parties can exchange device status;

[0092] The lender broadcasts the device status to notify the borrower;

[0093] The borrower's user software initiates a borrowing request for a device, establishes communication with the lender, intercepts the configuration request, and sends it to the lender through an RPC call;

[0094] The borrower injects the device into the local PCIe subsystem, creates a virtual PCIe device locally, and the Linux system loads the corresponding device driver;

[0095] After the device driver is loaded, the borrower has control over the device. The borrower's host can use the borrowed device just like a local device, and the software is unaware of it.

[0096] The borrower gets control of the device and updates the device status to "in use". At this time, other hosts are unavailable.

[0097] After the borrower has finished using the device, the device resources are released, the lender is notified, and the device status is updated to available;

[0098] Other hosts can now borrow the device again.

[0099] In this embodiment, each node in the host cluster registers an event monitoring callback function with the bottom layer when loading the RPC remote system for initialization. When the bottom layer NTB detects the node joining and failure exiting, it notifies the upper layer through the event mechanism and performs corresponding operations according to the event type. When a node joining and exiting event occurs, the bottom layer topology system notifies all nodes in the system of the event.

[0100] When the event type is EVT_HOT_ADD, a handshake message is sent to the node through the node ID fed back by the underlying layer (that is, the newly added node ID). After the new node receives the handshake message, it initializes the relevant structure in the message callback function and sends a reply message. At this time, the handshake connection between the newly added node and the local node is completed.

[0101] When the event type is EVT_HOT_REMOVE, first traverse all instance instances (each instance represents a node), and determine whether the node has a connection relationship with the faulty node based on the instance id. If there is no corresponding instance id, it means that the connection no longer exists, and you can exit directly. When there is a connection relationship, corresponding resource recovery processing is required. Specifically, if the node borrows the device of the faulty node, it is necessary to actively return it and remove the device resources from the remote device list; if the device of the node is used by the faulty node, it is necessary to actively recycle the device resources and put the device back into the resource pool for other nodes in the system to continue to use.

[0102] This embodiment also proposes a heterogeneous computing system device sharing device based on PCIe Switch, which is used to share PCIe devices in an interconnected system among all nodes. The nodes are divided into borrowers and lenders according to the borrowing relationship. The nodes have a borrowing module, a lending module and a node event dynamic monitoring module.

[0103] The borrowing module is used to manage the PCIe device belonging to the lender, including obtaining the PCIe device status of the lender, obtaining the PCIe device configuration information of the lender when the PCIe device belonging to the lender can be borrowed, creating a virtual PCIe device based on the obtained configuration information, accessing the PCIe device belonging to the lender through the virtual PCIe device, and returning and releasing the PCIe device resources belonging to the lender after exiting the access;

[0104] The lending module is used to manage the local PCIe device, including notifying the borrower of the local PCIe device status and reclaiming the lent PCIe device resources after the borrower quits accessing the PCIe device;

[0105] The node event dynamic monitoring module is used to monitor the node status in the interconnected system, establish a handshake connection with the newly added node, determine whether there is a connection relationship with the exiting node, and perform processing including exiting the connection or recovering the loaned PCIe device resources based on the judgment result.

[0106] Optionally, in the event of node joining, the local node sends a handshake message to the newly joined node through the newly joined node ID fed back by the underlying layer. After receiving the handshake message, the newly joined node returns a reply message in the message callback function to form a handshake connection between the newly joined node and the local node.

[0107] Optionally, in the event of a node exit, the local node first traverses all instance instances and determines whether the node has a connection relationship with the faulty node based on the instance id. If there is no corresponding instance id, it means that the connection no longer exists. In this case, it can exit directly. If there is a connection relationship, resource recovery processing will be performed.

[0108] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A heterogeneous computing system device sharing device based on PCIe Switch, used to share PCIe devices in an interconnected system among all nodes, wherein the nodes are divided into borrowers and lenders according to the borrowing relationship, characterized in that: The node has a borrowing module, a lending module and a node event dynamic monitoring module; The borrowing module is used to manage the PCIe device belonging to the lender, including obtaining the PCIe device status of the lender, obtaining the PCIe device configuration information of the lender when the PCIe device belonging to the lender can be borrowed, creating a virtual PCIe device based on the obtained configuration information, accessing the PCIe device belonging to the lender through the virtual PCIe device, and returning and releasing the PCIe device resources belonging to the lender after exiting the access; The lending module is used to manage the local PCIe device, including notifying the borrower of the local PCIe device status and reclaiming the lent PCIe device resources after the borrower quits accessing the PCIe device; The node event dynamic monitoring module is used to monitor the node status in the interconnected system, establish a handshake connection with the newly added node, determine whether there is a connection relationship with the exiting node, and perform processing including exiting the connection or recovering the loaned PCIe device resources according to the judgment result; In the event of a node joining, the local node sends a handshake message to the newly joined node through the newly joined node ID fed back by the underlying layer. After receiving the handshake message, the newly joined node returns a reply message in the message callback function to form a handshake connection between the newly joined node and the local node. In the event of a node exit, the local node first traverses all instance instances and determines whether the node has a connection relationship with the faulty node based on the instance id. If there is no corresponding instance id, it means that the connection no longer exists. In this case, it can exit directly. If there is a connection relationship, resource recycling will be performed.

2. A device sharing method for a heterogeneous computing system based on PCIe Switch, characterized in that: The method is implemented by the heterogeneous computing system device sharing device based on PCIe Switch as claimed in claim 1, and comprises the following steps: The lender and the borrower send messages to each other, establish a handshake connection, and create an instance of the other end; The lender adds the sharable PCIe device to the device pool and sets the sharable PCIe device status to a usable status; The lender actively connects to the borrower and creates a shared memory area so that both parties can exchange PCIe device status; The lender broadcasts the PCIe device status to notify the borrower; The borrower initiates a borrowing request for a PCIe device, and the lender returns the configuration information of the PCIe device; The borrower creates a virtual PCIe device based on the configuration information returned by the lender, and the borrower's access to the virtual PCIe device is transparently forwarded to the lender's PCIe device; After the borrower obtains the control right of the PCIe device belonging to the lender, the lender marks the status of the PCIe device as being in use and prevents other nodes in the interconnected system from accessing the PCIe device; After the borrower quits accessing the PCIe device belonging to the lender, the lender marks the status of the PCIe device as usable, and other hosts in the interconnected system can borrow the PCIe device.

3. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: When the borrower initiates a borrowing request for a PCIe device, the borrower actively intercepts the configuration read and write methods of the PCIe device, and transfers the configuration read and write requests of the PCIe device to the lender through RPC remote calls, and the lender accesses the configuration space of the PCIe device.

4. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: When the borrower initiates a borrowing request for a PCIe device, the borrower determines the BDF of the PCIe device through the hook function. If the device being accessed is a remote device, an RPC remote call is initiated to complete the reading and writing of the actual configuration of the PCIe device. At the same time, the borrower maintains the configuration space of the PCIe device in the local memory buffer and reads directly from the local cache when a remote call is not required.

5. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: The configuration information includes configuration space parameters and memory address space parameters.

6. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: The configuration information of the virtual PCIe device is the same as the configuration information of the shared PCIe device.

7. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: The configuration read request interception process of the shared PCIe device includes the following steps: The borrower host issues a read request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration read method is used. If it is a shared device, the borrower initiates an RPC remote call to notify the lender host to complete the read operation of the device configuration space. The lender host reads the data and returns the result to the borrower host. The borrower host saves the return value in the local cache.

8. The method for sharing heterogeneous computing system devices based on PCIe Switch according to claim 2, characterized in that: The configuration write request interception process of the shared PCIe device includes the following steps: The borrower host issues a write request to access the configuration space of the shared device. In the previously registered hook function, the BDF number of the device is used to determine whether the device is a shared device. If it is not a shared device, the original configuration write method is used. If it is a shared device, the data is first written to the local cache, and then the borrower initiates an RPC remote call to notify the lender host to complete the write operation of the device configuration space. The lender host receives the request and writes the data to the device's configuration space.

Images