RDMA application IO performance monitoring method, device, system and equipment

Abstract

The invention provides an RDMA (Remote Direct Memory Access) application IO (Input / Output) performance monitoring method, device, system and equipment, which are used for solving the technical problem of lack of a special RDMA application IO performance counting and monitoring system. According to the technical scheme provided by the invention, extra hardware support is not needed, only the user mode driver library provided by an RDMA network card manufacturer needs to be improved, IO performance related data of the RDMA application is obtained at the WR issuing interface and the polling interface, and an access interface is provided for the outside through a user mode performance statistical service; therefore, the purpose of monitoring the performance of the RDMA application driven by the RDMA network card is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data communication technology, and in particular to a method, device, system and equipment for monitoring RDMA application IO performance. Background Art

[0002] With the rapid development of high-performance computing (HPC), big data processing and data centers, the requirements for network communication performance are getting higher and higher. Traditional networks require the participation of the operating system kernel to perform protocol stack processing, context switching, high-frequency interrupt processing and arbitration of network resources, which brings high transmission delays and huge CPU consumption, and can no longer meet the requirements of high-performance networks. In order to improve the performance and scalability of network equipment, more and more manufacturers have begun to adopt user-mode driven Remote Direct Memory Access (RDMA) technology, which has been widely used in various fields mentioned above due to its low latency, high bandwidth and low overhead.

[0003] However, while providing high-performance networks, RDMA also brings some challenges. Due to its unique architectural design, data is directly transferred from the memory area of ​​the user-mode application to the network card. Since it does not pass through the kernel, although the number of memory copies and overhead are reduced, it also makes it impossible to accurately count in the kernel state. If users need performance statistics functions to complete bottleneck analysis, they need to design them separately every time they develop RDMA applications. In addition, some key data structures in the InfiniBand Verbs (IBV) protocol library are invisible to upper-layer applications, making it difficult to implement some statistical details.

[0004] Currently, the Open Fabrics Enterprise Distribution (OFED) driver packages and related open source projects provided by RDMA network card manufacturers only provide some relatively scattered simple supporting tools for performance testing and bandwidth viewing. Moreover, these tools can only perform flow testing or view the statistics of sent and received packets for RDMA network cards, but cannot effectively monitor IO performance data for complex user scenarios.

[0005] In short, in terms of RDMA user-mode input / output (IO) flow performance data monitoring, some existing tools and methods are difficult to meet some high-precision performance analysis and optimization requirements. For example, according to the number of packets sent per second, bandwidth and IO latency of each RDMA user application, to analyze its performance and bottlenecks, or to monitor the traffic fluctuations of RDMA applications to find the cause of the problem. Summary of the invention

[0006] In view of this, the present invention provides a method, apparatus, system and device for monitoring IO performance of RDMA applications, which are used to solve the technical problem of lack of IO performance statistics and monitoring system specifically for RDMA applications.

[0007] Based on one aspect of the present invention, the present invention provides an RDMA application IO performance monitoring method, which is applied to a device that deploys a remote direct memory access RDMA network card and an RDMA application, wherein the RDMA application calls a user-mode driver library through a wireless bandwidth transmission IBVerbs standard library, and the user-mode driver library establishes a QP link with a peer device through the RDMA network card based on a queue pair QP. The method comprises:

[0008] The interface for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion WC information from the completion queue CQ in the user-mode driver library to obtain IO performance related data of the RDMA application;

[0009] The user-state performance statistics service in the user-state driver library provides an access interface for RDMA application IO performance related data.

[0010] Furthermore, the RDMA application loads and runs the IB Verbs standard library and the user-mode driver library in its own process space when it is started. The user-mode performance statistics service in the user-mode driver library provides an interface for accessing IO performance related data of the RDMA application based on the unique identifier of the RDMA application.

[0011] Furthermore, the IO performance related data of the RDMA application includes one or more of the following data:

[0012] Statistical time, application IO processing rate per second, actual NIC IO processing rate per second, total number of IO sent by the application, actual total number of IO sent by the NIC, application data sending rate, actual NIC data sending rate, SQ queue status, work request WR delay, WR length, total number of WR sent by the application.

[0013] Furthermore, when counting the total number of IOs sent by the application, the total number of IOs actually sent by the network card, WR latency, application data sending rate, and actual data sending rate of the network card, according to the situation where WR carries the signal mark ib sig, it is divided into the normal situation where the last WR sent each time must carry ib sig and the abnormal situation where ib sig is only carried once in multiple transmissions, and statistics are performed for the two situations separately.

[0014] Based on another aspect of the present invention, the present invention further provides an RDMA application IO performance monitoring device, which is applied to a device deploying an RDMA network card and an RDMA application, wherein the RDMA application calls a user-mode driver library through a wireless bandwidth transmission IB Verbs standard library, and the user-mode driver library establishes a QP link with a peer device through the RDMA network card based on a queue pair QP. The device comprises:

[0015] The data acquisition module is located in the user-mode driver library, and the interface in the user-mode driver library for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion information WC from the completion queue CQ obtain the IO performance related data of the RDMA application;

[0016] The user-state performance statistics service unit is located in the user-state driver library and is used to provide an access interface for RDMA application IO performance related data to the outside world.

[0017] Furthermore, the RDMA application loads and runs the IB Verbs standard library and the user-mode driver library in its own process space when it is started;

[0018] The user state performance statistics service unit provides an interface for accessing IO performance related data of the RDMA application based on the unique identifier of the RDMA application.

[0019] Furthermore, the IO performance related data of the RDMA application includes one or more of the following data:

[0020] Statistical time, application IO processing rate per second, actual NIC IO processing rate per second, total number of IO sent by the application, actual total number of IO sent by the NIC, application data sending rate, actual NIC data sending rate, SQ queue status, work request WR delay, WR length, total number of WR sent by the application.

[0021] Based on one aspect of the present invention, the present invention also provides an RDMA application IO performance monitoring system, the system comprising:

[0022] RDMA application, used to transmit IB Verbs standard library calls user-mode driver library through wireless bandwidth, and establish QP link with peer device through RDMA network card based on queue pair QP;

[0023] IB Verbs standard library, used to provide a unified abstract interface for RDMA applications;

[0024] The user-mode driver library is used to provide a specific implementation interface for the unified abstract interface in the IB Verbs standard library, and to connect with the unified abstract interface of the IB Verbs standard library; the interface in the user-mode driver library for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion WC information from the completion queue CQ to obtain IO performance related data of the RDMA application;

[0025] RDMA network card, used to provide data transmission based on RDMA protocol for user-mode driver library;

[0026] User-state performance statistics service, located in the user-state driver library, is used to provide an access interface for RDMA application IO performance related data;

[0027] Client tool used to obtain IO performance data of RDMA applications from the user-mode performance statistics service.

[0028] Furthermore, the system further comprises:

[0029] System monitoring and alarm tools, used to connect to multiple client tools, obtain and store IO performance related data of multiple RDMA applications through multiple client tools;

[0030] The data visualization and monitoring platform is used to connect to system monitoring and alarm tools to provide users with a visual presentation of IO performance-related data of RDMA applications.

[0031] The technical solution provided by the present invention does not require additional hardware support, but only requires improvement of the user-state driver library provided by the RDMA network card manufacturer, obtains IO performance-related data of the RDMA application through the WR sending interface and the polling interface, and provides an access interface to the outside through the user-state performance statistics service, thereby achieving the purpose of performance monitoring of the RDMA application driven by the RDMA network card. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.

[0033] Figure 1 A schematic diagram of the RDMA driver framework structure is provided for an embodiment of the present invention;

[0034] Figure 2A schematic diagram of a system structure for implementing RDMA application IO performance monitoring in one embodiment of the present invention;

[0035] Figure 3 A schematic diagram of the steps of a method for monitoring IO performance of an RDMA application provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the latency associated with RDMA application IO performance in one embodiment of the present invention;

[0037] Figure 5 It is a schematic diagram of the delay statistics of RDMA application sending and polling WR under normal circumstances in one embodiment of the present invention;

[0038] Figure 6 A schematic diagram of the delay statistics of RDMA application sending and polling WR under unconventional circumstances in one embodiment of the present invention;

[0039] Figure 7 It is a schematic diagram of a method for collecting and presenting RDMA application IO performance data based on a client tool in one embodiment of the present invention;

[0040] Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present invention for implementing the RDMA application IO performance monitoring method provided in the present invention. DETAILED DESCRIPTION

[0041] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. Although the embodiments of the present invention may be described in a first, second, or third manner, this description is only used to distinguish similar information, entities, or steps, rather than to describe a specific order or sequence. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. For another example, in some scenarios, the first information may refer to one information or to multiple information of the same type. In addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination". The "and / or" in the present invention is only a kind of association relationship describing the associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B may be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0042] The purpose of the present invention is to provide a real-time, efficient and accurate RDMA application IO performance monitoring solution for high-performance computing and distributed systems in RDMA scenarios, to help users deeply understand the IO performance and operation of applications (hereinafter referred to as RDMA applications) that support RDMA protocol network cards, to provide strong support for performance optimization and problem solving, and to solve the problems of hardware dependence, performance loss and few statistical dimensions in the current solutions for RDMA application performance statistics. The technical solution provided by the present invention does not require additional hardware support, but only requires improving the user-mode driver library provided by the RDMA network card manufacturer, and then adapting a client tool to achieve the purpose of performance monitoring of all RDMA applications driven by this network card.

[0043] The specific implementation process of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the step processes shown in the accompanying drawings and embodiments can be executed in a computer system such as a set of computer executable instructions, and one or more of the exemplified method steps can be single or multiple or aggregated together and run in the computer system in the form of a software module or a virtual device. Moreover, although the exemplified step processes have time and sequential logic, the present invention is not limited to these steps in certain cases. They can be executed in an order different from that here. Under the guidance of the purpose of the present invention, simple changes or combinations of the execution order of these steps should also fall within the scope of the disclosure of the embodiments of the present invention.

[0044] Figure 1 A schematic diagram of the RDMA driver framework structure is provided for an embodiment of the present invention. A host (such as machine 1 or machine 2) includes three layers: hardware, kernel state and user state. In the user state layer, the Infiniband IB Verbs standard library (such as libibverbs, librdmacm, etc.) provides a unified abstract interface to the upper-layer application that supports the RDMA protocol. The user state driver provided by the network card manufacturer provides a specific implementation interface (such as libmlx5.so) corresponding to the unified abstract interface in the standard library to implement the connection with the unified abstract interface of the IB Verbs standard library. In the kernel state, the kernel state driver (such as mlx5_core.ko) processes some commands issued by the user state driver through the ioctl method, and the kernel state driver is connected to the RDMA network card control of the hardware layer.

[0045] The interaction relationship and process of the entire RDMA driver architecture are as follows: the RDMA application calls the IB Verbs standard library interface, and according to the model of the current network card, sends commands to the corresponding kernel module through the user-mode driver interface provided by the manufacturer attached to the interface to configure resources and establish a connection with the peer device based on a queue pair (Queue Pair, QP). Then the RDMA application uses the interface provided by the user-mode IB Verbs standard library to send work requests (WorkRequest, WR) to the send queue (SQ) (for example, ibv_post_send), to send work requests WR to the receive queue (Receive Queue, RQ) (for example, ibv_post_recv), and to poll the completion queue (Completion Queue, CQ) for work completion. The interface for receiving completion (WC) information (such as ibv_poll_cq, referred to as the polling interface) directly sends data to the network card hardware for packet sending or directly obtains data from the network card. It no longer needs to be triggered by an interrupt like the traditional network mode, and it also needs to go through the kernel network protocol stack and multiple memory copies to complete the process of sending data to the network card.

[0046] The RDMA driver has the following characteristics: the management flow and data flow are separated; the data flow goes directly to the RDMA network card without passing through the kernel space; the network protocol is fixed in the hardware, and the encapsulation and parsing of data packets are performed in the network card firmware; only a standard interface library is provided for upper-layer applications.

[0047] Figure 2 The figure is a schematic diagram of the system structure for implementing RDMA application IO performance monitoring in one embodiment of the present invention.

[0048] The present invention is based on the RDMA driver framework, and obtains performance-related data at the key interface position of the IO stream reading, writing, receiving, sending, and other operations in the user state, and performs statistical counting on the obtained performance-related data (setting marks and statistical counting can be understood as adding counting, timestamp recording, and program code for obtaining performance-related data functions at key interface positions). Through the newly added performance statistics service in the user state to provide configuration interfaces, statistical data output interfaces, and other interfaces, the client tool can configure the performance statistics service of the specified RDMA application, obtain statistical data from the performance statistics service, and present it.

[0049] like Figure 2 In the example, after the RDMA application establishes a queue-to-QP link through the InfiniBand IB Verbs standard library interface, it encapsulates the work request WR and sends the WR to the corresponding queue for data transmission and reception. The main processing flow is as follows:

[0050] Sending process: WR is sent to the SQ ring sending queue through the interface (ibv_post_send) that sends work requests to the sending queue. The work request WR will be converted into a work queue element (WQE), and then the network card firmware will be notified to extract the WQE from the SQ for processing (disassembling the data in the sending memory into packets and sending them out).

[0051] Receiving process: WR is sent to the RQ ring receiving queue through the interface (ibv_post_recv) that sends work requests to the receiving queue. The work request WR will be converted into a work queue element WQE, and then the network card firmware will be notified to extract WQE from RQ for processing (unpack the packet and copy it to the corresponding receiving memory).

[0052] Polling process: poll the CQ ring completion queue through the interface for polling work completion information (ibv_poll_cq). After the RDMA network card completes the processing, a work completion WC event will be generated and put into the CQ queue. In this way, the interface for polling work completion information can poll the corresponding WR completion event. If the obtained WC completion event indicates that the work is successfully completed, it means that an IO request has been completed.

[0053] Figure 3 A schematic flow chart of the steps of an RDMA application IO performance monitoring method provided by an embodiment of the present invention is provided. The method is applied to a device where an RDMA network card and an RDMA application are deployed. The RDMA application on the device calls the user-mode driver library improved by the present invention through the wireless bandwidth transmission IBVerbs standard library docking. The user-mode driver library establishes a QP connection with the peer device through the RDMA network card based on the queue pair QP. The method includes:

[0054] S301. An interface for sending work requests WR to a send queue SQ, an interface for sending work requests WR to a receive queue RQ, and an interface for polling work completion information from a completion queue CQ WC to obtain IO performance-related data for RDMA applications in a user-mode driver library;

[0055] The IO performance related data of the RDMA application described in the present invention refers to the related data or statistical results obtained in the user-mode driver library that can reflect the performance of the RDMA application performing data IO through the RDMA network card. Each RDMA application process will load the IB Verbs standard library and the improved user-mode driver library of the present invention in its own process space when it is loaded and run. In the user-mode driver library, a performance statistics service thread can be started separately to monitor the UNIX domain socket named with the RDMA application process ID to respond to commands from the client tool, thereby realizing the maintenance, sorting or statistics of the IO performance related data of the specific RDMA application and returning the results.

[0056] In one embodiment of the present invention, the RDMA application IO performance related data may be stored in a data structure built into the RDMA link queue pair QP memory context. The RDMA application IO performance related data may be obtained by improving interfaces such as ibv_post_send, ibv_post_recv, and ibv_poll_cq, adding variables, counters, and acquisition codes for obtaining performance related data.

[0057] The present invention does not limit the storage location of the performance-related data and statistical data, and the performance-related data and statistical data may be cached in a local memory or stored in a local file or database, as long as they can be acquired and processed by the performance statistics service.

[0058] S302: Provide an access interface for RDMA application IO performance related data through the user state performance statistics service in the user state driver library.

[0059] The present invention adds a user state performance statistics service in the user state driver, and the user can perform maintenance, sorting, statistics and other related processing on the acquired RDMA performance related data through the interface provided by the user state performance statistics service.

[0060] After running, each RDMA application will generate a unique process ID corresponding to the RDMA application. In order to provide the IO performance related data of the RDMA application to the outside, in one embodiment of the present invention, when the user state performance statistics service is initialized, a service interface is started. The name of the service interface includes the RDMA application process ID. The client tool can access the service interface based on the RDMA application process ID and access the RDMA performance related data in the performance statistics service through the service interface. The service interface can be a socket listening interface, an HTTP interface, etc. The present invention does not limit the specific interface type.

[0061] Client tool connected to the performance statistics service: command-line-based tool that sends commands to the performance statistics service for processing through the UNIX domain socket file. It supports turning on or off the statistics function, printing the queried statistics information to the terminal, clearing and resetting, etc.

[0062] In one embodiment of the present invention, obtaining IO performance related data of the RDMA application by the interface for sending WR information and the interface for polling WC information in the user-mode driver library includes obtaining one or more of the following IO performance related data:

[0063] Statistical time: refers to the time interval from enabling performance monitoring to executing a query. The present invention allows a user to enable the IO performance monitoring function of a certain RDMA application through a terminal tool command line. The statistical time refers to the time interval from enabling performance monitoring to executing a query.

[0064] Application IO processing rate per second (Input / Output Operations Per Second, IOPS): refers to the IO sending capability of the upper-layer RDMA application.

[0065] NIC actual IO processing rate per second (IOPS): refers to the actual IO sending capability of the RDMA NIC.

[0066] Total number of IOs sent by applications: refers to the total number of IOs sent by upper-layer RDMA applications.

[0067] Total number of IOs actually sent by the network card: refers to the total number of IOs actually processed by the RDMA network card.

[0068] Application data sending rate: refers to the sending bandwidth of the RDMA application.

[0069] Actual data sending rate of the network card: refers to the actual sending bandwidth of the RDMA network card.

[0070] Send queue SQ queuing status: refers to the real-time length of the send queue SQ in the queue pair QP.

[0071] WR delay: refers to the time from the issuance of a work request WR to its completion, which may include maximum, minimum, and average values.

[0072] WR length: refers to the data size carried by the work request WR.

[0073] Total number of WRs sent by the application: refers to the total number of work request WRs sent.

[0074] Figure 4The figure is a schematic diagram of the latency related to the IO performance of RDMA applications in one embodiment of the present invention. In the figure, T_1, T_2, T_3, T_4, T_5, and T_6 correspond to the latency of different parts related to IO performance. The sum of T1 to T6 is the entire process from calling the work request issuing interface, including hardware processing, receiving the ACK response, to finally confirming that the work request WR has been processed, that is, the latency of a complete work request WR.

[0075] Figure 5 The figure is a schematic diagram of the delay statistics of RDMA application sending and polling WR under normal circumstances in one embodiment of the present invention. Among them, △t1 is the delay between sending work request WR1 and polling work completion information WC1 of WR1, and △t2 is the delay between sending work request WR2 and polling work completion information WC2 of WR2.

[0076] Figure 6 This is a schematic diagram of the latency statistics of RDMA application sending and polling WR under unconventional circumstances in one embodiment of the present invention. RDMA data sending and receiving is based on the asynchronous implementation of queue pairs. The two axes represent calling different interfaces to operate different queues. SEND represents sending WR, and POLL represents polling WC events.

[0077] The following is an example of how to obtain some data based on the above method of obtaining IO performance related data:

[0078] 1) Total number of IOs sent by the application: This can be counted at the ibv_post_send interface and accumulated into the data structure built into the queue pair QP memory context of the RDMA link.

[0079] 2) The total number of IOs actually sent by the network card: To count this value, it is necessary to divide it into two cases, namely:

[0080] a) Normal situation (the last WR sent each time must carry ib sig)

[0081] Under normal circumstances, when the work request WR is sent post send, the index idx corresponding to the last WR carries IBV_SEND_SIGNALED, i.e., the signal tag ib sig. The ib sig tag is a special value of the WR send tag. Only the WR with this tag will return the corresponding WC event after the network card processes it. When the WR is polled, the ib sig tag is detected, and the number of IOs that are actually sent out and polled to complete information is updated to the corresponding QP memory context statistical data structure.

[0082] b) Unconventional situation (sent multiple times, only one with ib sig):

[0083] In unusual circumstances, when a work request WR is sent post send, for a WR that does not carry ibsig in a single send, it is added to the corresponding WR tag with ib sig that follows, and is updated in the same manner as above during poll.

[0084] 3) Total number of WRs sent by the application: Count at the post send interface, and add all the WRs of that time to the statistical structure of the corresponding QP memory context at the last WR of each time.

[0085] 4) WR length: count at the ibv_post_send interface and aggregate the length in WR to the statistics structure of the corresponding QP memory context.

[0086] 5) WR delay: Figure 4 As shown in the figure, the WR delay is the sum of T_1, T_2, T_3, T_4, T_5, and T_6. The WR delay is obtained by tracing the complete path of the specified WR and summing up all the times. The main delays in the WR delay are T_5 and T_6.

[0087] T_5 is mainly because the delay here is unpredictable. It depends on the delay of the local hardware sending to the peer end, the response of the peer end, the quality of the physical link, etc.

[0088] T_6 has a larger delay mainly due to the upper-layer scheduling factors.

[0089] In addition, latency statistics are divided into the following two situations:

[0090] Normal situation (each WR carries ib sig): when sending, record the current time (Ts1) and save it to the corresponding idx space on the SQ chain. After the corresponding idx is polled, record the polled time (Tc1). Tc1-Ts1 is the corresponding delay, which indicates the complete time required for a WR to be sent from the upper layer to the hardware, and then aggregated to the statistical structure of the corresponding QP memory context.

[0091] Unconventional situation (only one of multiple transmissions carries ib sig or only the last one carries ibsig): the recorded transmission time is different from the normal one. Ts1 records the time from the previous transmission with ib sig to the next transmission without ib sig, and transmits it to the next idx with ib sig. At this time, the delay is the sum of the time of the current n WRs.

[0092] 6) Application data sending rate: By tracking the path from the WR sent by the upper-layer application to the delivery hardware, once the delivery hardware is processed, it means that the upper-layer sending process is completed, and the length of the information carried by each WR is obtained. By accumulating the WRs per unit time, the number of WRs sent by the upper layer, the sending rate of the amount of information carried by the WRs, and other information can be obtained. At the ibv_post_send interface, the current total number of sent WRs is accumulated and updated to the temporary count of the QP context. Under normal circumstances, these counts will be saved in the idx corresponding to the WR with ib sig. After the corresponding WR is polled, the current total number of sends - the total number of sends saved in idx = the total number of sends in the current time period; according to the time difference calculated above, the total number of sends in the current time period × the average length of WR ÷ the time difference = the instantaneous upper-layer application data sending rate (unconventional situations are similar to the previous processing delay).

[0093] 7) Actual data transmission rate of the network card:

[0094] It is necessary to consider that the CQE generated by the hardware after a WR is sent out and receives an ACK response from the other end can be regarded as a complete sending process, so it is necessary to record the whole process from the sending WR to Poll (polling, if no normal return is found, wait a while and send the query again) to CQE per unit time, and the statistical points are: the number of CQEs polled, the id of the WR matched by CQE, the information length of WR, etc. At the ibv_poll_cq interface, the current total number of polls to WR is accumulated and updated to the temporary count of the QP context. Under the above-mentioned normal circumstances, these counts will be saved in the idx corresponding to the WR with ib sig. After the corresponding WR is polled, the current total number of polls - the total number of polls saved in idx = the total number of polls in the current time period; according to the time difference calculated above, the total number of polls in the current time period × the average length of WR ÷ the time difference = the instantaneous actual sending rate (unconventional situations are similar to the delay processing above).

[0095] 8) Sending queue SQ queue status: the number of WRs that have not been sent in the SQ queue of the current QP is accumulated to the corresponding QP statistics structure.

[0096] Figure 7The present invention is a schematic diagram of the method of collecting and presenting RDMA application IO performance data based on client tools in one embodiment of the present invention. RDMA Exporter-1 to RDMA Exporter-X in the figure are client tools, corresponding to RDMA application-1 to RDMA application-X respectively. After each RDMA application instantiates, loads and runs the user-state driver, the RDMA application IO performance data monitoring method provided by the present invention can generate statistical data through the improved user-state driver. The client tool, such as RDMA Exporter-1, connects to the performance statistics service in the user-state driver of the application through the unique identifier of RDMA application-1, such as the process identifier ID, so as to obtain statistical data related to the RDMA application IO performance.

[0097] After obtaining statistical data, the client tool can transmit the data to the distributed system monitoring and alarm tool (such as Prometheus) for storage and further processing, and then present it to the user through the data visualization and monitoring platform (Grafana), so that the user can view the performance data of the RDMA application of interest in real time on the WEB page.

[0098] Figure 8 The electronic device structure diagram for implementing the RDMA application IO performance monitoring method provided by the present invention is provided in an embodiment of the present invention. The device 800 includes: a processor 810 such as a central processing unit (CPU), a communication bus 820, a communication interface 840, and a memory 830. The processor 810 and the memory 830 can communicate with each other through the communication bus 820. The memory 830 stores a computer program, and when the computer program is executed by the processor 810, the function of one or more steps in the RDMA application IO performance monitoring method provided by the present invention can be implemented.

[0099] Memory refers to a device based on a certain storage medium for storing computer programs and / or data. It can be a volatile memory (VM, often called memory) or a non-volatile memory (NVM). Memory refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It is used as a storage medium for temporary data of the operating system and other running programs. Memory can be synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), etc. Non-volatile memory refers to a memory that uses a persistent storage medium. It has the characteristics of large capacity and persistent data storage. It can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. SCM is the industry's general term for new storage media between memory and flash memory. It is a composite storage technology that combines persistent storage characteristics and memory characteristics. Its access speed is slower than DRAM and faster than SSD hard disks.

[0100] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0101] It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium configured in this way causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0102] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0103] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring RDMA application IO performance, characterized in that: The method is applied to a device that deploys a remote direct memory access (RDMA) network card and an RDMA application. The RDMA application calls a user-mode driver library through a wireless bandwidth transmission IB Verbs standard library. The user-mode driver library establishes a QP link with a peer device through the RDMA network card based on a queue pair QP. The method includes: The interface for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion WC information from the completion queue CQ in the user-mode driver library to obtain IO performance related data of the RDMA application; The user-state performance statistics service in the user-state driver library provides an access interface for RDMA application IO performance related data.

2. The method according to claim 1, characterized in that: The RDMA application loads and runs the IB Verbs standard library and the user-mode driver library in its own process space when it is started. The user-mode performance statistics service in the user-mode driver library provides an interface for accessing IO performance related data of the RDMA application based on the unique identifier of the RDMA application.

3. The method according to claim 1, characterized in that: The IO performance related data of the RDMA application includes one or more of the following data: Statistical time, application IO processing rate per second, actual NIC IO processing rate per second, total number of IO sent by the application, actual total number of IO sent by the NIC, application data sending rate, actual NIC data sending rate, SQ queue status, work request WR delay, WR length, total number of WR sent by the application.

4. The method according to claim 2, characterized in that When counting the total number of IOs sent by the application, the total number of IOs actually sent by the network card, WR latency, application data sending rate, and actual data sending rate of the network card, according to the situation where WR carries the signal mark ib sig, it is divided into the normal situation where the last WR sent each time must carry ib sig and the abnormal situation where ib sig is carried only once among multiple transmissions. Statistics are collected for the two situations respectively.

5. A RDMA application IO performance monitoring device, characterized in that: The device is applied to a device that deploys an RDMA network card and an RDMA application. The RDMA application calls a user-mode driver library through a wireless bandwidth transmission IB Verbs standard library. The user-mode driver library establishes a QP link with a peer device through the RDMA network card based on a queue pair QP. The device includes: The data acquisition module is located in the user-mode driver library, and the interface in the user-mode driver library for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion information WC from the completion queue CQ obtain the IO performance related data of the RDMA application; The user-state performance statistics service unit is located in the user-state driver library and is used to provide an access interface for RDMA application IO performance related data to the outside world.

6. The device according to claim 5, characterized in that The RDMA application loads and runs the IB Verbs standard library and the user-mode driver library in its own process space when it is started; The user state performance statistics service unit provides an interface for accessing IO performance related data of the RDMA application based on the unique identifier of the RDMA application.

7. The device according to claim 5, characterized in that The IO performance related data of the RDMA application includes one or more of the following data: Statistical time, application IO processing rate per second, actual NIC IO processing rate per second, total number of IO sent by the application, actual total number of IO sent by the NIC, application data sending rate, actual NIC data sending rate, SQ queue status, work request WR delay, WR length, total number of WR sent by the application.

8. An RDMA application IO performance monitoring system, characterized in that: The system includes: RDMA application, used to transmit IB Verbs standard library calls user-mode driver library through wireless bandwidth, and establish QP link with peer device through RDMA network card based on queue pair QP; IB Verbs standard library, used to provide a unified abstract interface for RDMA applications; The user-mode driver library is used to provide a specific implementation interface for the unified abstract interface in the IB Verbs standard library, and to connect with the unified abstract interface of the IB Verbs standard library; the interface in the user-mode driver library for sending work requests WR to the sending queue SQ, the interface for sending work requests WR to the receiving queue RQ, and the interface for polling work completion WC information from the completion queue CQ to obtain IO performance related data of the RDMA application; RDMA network card, used to provide data transmission based on RDMA protocol for user-mode driver library; User-state performance statistics service, located in the user-state driver library, is used to provide an access interface for RDMA application IO performance related data; Client tool used to obtain IO performance data of RDMA applications from the user-mode performance statistics service.

9. The system according to claim 8, characterized in that The system further comprises: System monitoring and alarm tools, used to connect to multiple client tools, obtain and store IO performance related data of multiple RDMA applications through multiple client tools; The data visualization and monitoring platform is used to connect to system monitoring and alarm tools to provide users with a visual presentation of IO performance-related data of RDMA applications.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium communicate with each other via the communication bus; Storage medium for storing computer programs; A processor, for implementing the method according to any one of claims 1 to 4 when executing a computer program stored on a storage medium.

Images