Anomaly detection method and apparatus for RDMA system, electronic device, and readable medium

Abstract

The present disclosure provides an anomaly detection method and apparatus for an RDMA system, an electronic device, and a readable medium. The anomaly detection method for an RDMA system comprises: performing abstract modeling on the working space of an RDMA system to generate a search space of the RDMA system; determining workloads on the basis of the dimensions of the search space; determining conditional features of any workload that triggers an RDMA system anomaly, so as to generate a minimum functional feature set of the anomaly; and for any minimum functional feature set, calling a counter of the RDMA system to determine an anomaly index of the workload. By means of the embodiments of the present disclosure, active detection of performance anomalies in RDMA systems is achieved, improving the performance and stability of RDMA systems in practical applications.

Description

Anomaly detection method, device, electronic device and readable medium of RDMA system

[0001] The present disclosure claims priority to Chinese patent application number 202410064360.1 filed on January 16, 2024, entitled “Abnormality detection method, device, electronic device and readable medium for RDMA system”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of communication technology, and in particular to an abnormality detection method, device, electronic device, and readable medium for an RDMA system. Background Art

[0003] In related technologies, RDMA hardware vendors conduct extensive testing, but due to the large number of parties involved, RDMA systems may still experience various performance anomalies, seriously affecting RDMA network performance and stability.

[0004] However, existing RDMA performance testing mainly relies on simple benchmark tools or tests on known applications, which cannot fully cover potential application workloads and makes it difficult to detect performance anomalies.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0006] Summary of the Invention

[0007] The present disclosure aims to provide an RDMA system anomaly detection method, apparatus, electronic device, and readable medium, which are used to at least to some extent overcome the problem of poor reliability of the RDMA system caused by the limitations and defects of related technologies.

[0008] According to a first aspect of an embodiment of the present disclosure, a method for detecting anomalies in an RDMA system is provided, comprising: abstractly modeling a workspace of the RDMA system to generate a search space for the RDMA system; determining a workload based on a dimension of the search space; determining conditional features for triggering an anomaly in the RDMA system by any workload to generate a minimum functional feature set for the anomaly; and, for any minimum functional feature set, calling a counter of the RDMA system to determine an anomaly indicator of the workload.

[0009] In an exemplary embodiment of the present disclosure, it further includes:

[0010] The workload pattern and / or the workload selection result are adjusted based on the iterative calculation result of the abnormal indicator using a statistical algorithm.

[0011] In an exemplary embodiment of the present disclosure, adjusting the workload mode and / or the workload selection result based on the iterative calculation result of the abnormality indicator using a statistical algorithm includes:

[0012] Calling an annealing algorithm in the statistical algorithm to iteratively calculate a count difference between the count value of the workload and the count value of a new workload;

[0013] determining a ratio between the count difference and the count value of the workload;

[0014] The workload mode and / or the workload selection result are adjusted according to the ratio value.

[0015] In an exemplary embodiment of the present disclosure, adjusting the workload mode and / or the workload selection result according to the ratio value includes:

[0016] If it is determined that the ratio value is less than zero, transferring the workload to the new workload;

[0017] If it is determined that the proportional value is greater than zero, calculating the inverse of the proportional value using an exponential function with the natural constant e as the base;

[0018] A calculation result of the exponential function is determined as a probability of transferring the workload to the new workload.

[0019] In an exemplary embodiment of the present disclosure, adjusting the workload mode and / or the workload selection result according to the ratio value includes:

[0020] The request mode of the RDMA system and / or allocation information of the buffer of the RDMA system are adjusted according to the ratio value.

[0021] In an exemplary embodiment of the present disclosure, abstractly modeling the working space of the RDMA system to generate a search space of the RDMA system includes:

[0022] Abstracting the workspace of the RDMA system to determine a memory area corresponding to the workspace, and registering the memory area;

[0023] Creating the queue pair and setting the transmission type of the queue pair;

[0024] Create a work queue element based on the registered memory area and queue pair with the transfer type set;

[0025] The completion queue is created according to the work queue elements to generate a search space of the RDMA system.

[0026] In an exemplary embodiment of the present disclosure, determining the workload according to the dimension of the search space includes:

[0027] Parsing the dimensions of the search space including topology, host memory resources, transmission mode, and message mode;

[0028] Traffic between hosts of the RDMA system is generated according to the topology, the host memory source, the transmission mode, and the message mode, and the workload corresponding to the traffic is determined.

[0029] According to a second aspect of an embodiment of the present disclosure, there is provided an abnormality detection device for an RDMA system, comprising:

[0030] a modeling module configured to perform abstract modeling on the working space of the RDMA system to generate a search space of the RDMA system;

[0031] a determination module configured to determine a workload according to a dimension of the search space;

[0032] The determination module is configured to determine the conditional features of any workload triggering the RDMA system abnormality, so as to generate a minimum functional feature set of the abnormality;

[0033] The determining module is configured to call a counter of the RDMA system to determine an abnormality indicator of the workload for any of the minimum functional feature sets.

[0034] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the above methods based on instructions stored in the memory.

[0035] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the method for detecting an abnormality in an RDMA system as described in any one of the above is implemented.

[0036] The disclosed embodiments abstractly model the workspace of an RDMA system to generate a search space for the RDMA system. The workload is then determined based on the dimensions of the search space. The conditional features for triggering an RDMA system anomaly with any workload are determined to generate a minimum set of abnormal functional features. Finally, for any minimum set of functional features, the RDMA system's counters are called to determine workload anomaly indicators. This enables proactive detection of RDMA system performance anomalies and improves the performance and stability of the RDMA system in practical applications.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram showing an exemplary system architecture of an anomaly detection solution for an RDMA system to which an embodiment of the present invention may be applied;

[0039] FIG2 is a flow chart of an abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0040] FIG3 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0041] FIG4 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0042] FIG5 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0043] FIG6 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0044] FIG7 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0045] FIG8 is a flowchart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

[0046] FIG9 is a system architecture diagram of an anomaly detection solution for an RDMA system in an exemplary embodiment of the present disclosure;

[0047] FIG10 is a schematic diagram of a search space (workspace) of an anomaly detection solution for an RDMA system in an exemplary embodiment of the present disclosure;

[0048] FIG11 is a flowchart of an anomaly detection solution for an RDMA system in an exemplary embodiment of the present disclosure;

[0049] FIG12 is a block diagram of an abnormality detection device for an RDMA system in an exemplary embodiment of the present disclosure;

[0050] FIG13 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The described conditional features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0052] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0053] FIG1 is a schematic diagram showing an exemplary system architecture to which an anomaly detection solution for an RDMA system according to an embodiment of the present invention can be applied.

[0054] As shown in FIG1 , system architecture 100 may include one or more terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0055] It should be understood that the number of terminal devices, networks, and servers in FIG1 is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. For example, server 105 may be a server cluster consisting of multiple servers.

[0056] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0057] In some embodiments, the RDMA system anomaly detection method provided by the embodiments of the present invention is generally executed by the server 105. Accordingly, the RDMA system anomaly detection device is generally installed in the terminal device 103 (which may also be the terminal device 101 or 102). In other embodiments, certain terminals may have similar functions to the server device to execute the present method.

[0058] The technical terms involved in the embodiments of the present disclosure are explained and described below, specifically as follows:

[0059] MFS: Minimum Feature Set, minimum feature set or minimum feature set.

[0060] MR: Memory Region. Applications must first register a memory region to make it accessible to RDMA. Use this function to register an MR. An MR has a starting address and length, defining a contiguous memory space. The RNIC can directly access the registered MR without CPU involvement.

[0061] RNIC: RDMA Network Interface Card, RDMA network card.

[0062] QP: Queue Pair. Use ibv_create_qp to create a QP. A QP represents a "connection" between an application and an RNIC. A QP is an abstraction for point-to-point communication and includes a send queue and a receive queue. Each QP needs to be configured with a transmission mode, such as a reliable connection (RC).

[0063] WQE: Work Queue Element, work request. To send / receive messages, you need to construct a WQE and deliver it to the QP. The WQE contains a scatter / gather list that specifies a set of MR memory buffers participating in the transmission. The WQE is delivered to the QP sending queue.

[0064] Among them, scatter / gather is used to describe the operation of reading from or writing to the Channel.

[0065] Scatter: When reading from a Channel, the data read is written to multiple buffers during the read operation. Therefore, the Channel scatters the data read from the Channel into multiple buffers.

[0066] Gather: Writing to a Channel writes data from multiple buffers into the same Channel. Therefore, the Channel gathers the data from multiple buffers and sends it to the Channel. Scatter / gather is often used in situations where the transmitted data needs to be processed separately. For example, when transmitting data consisting of a message header and a message body, you can spread the message header and the message body into different buffers for convenient processing.

[0067] CQ: Completion Queue, completion queue, CQ is used to receive completion notifications and determine whether WQE is completed. Create CQ and query CQ. After obtaining the completion notification, the WQE memory buffer can be reused.

[0068] RC: Connection-oriented reliable service.

[0069] UC: Connection-oriented unreliable service.

[0070] UD: Datagram-oriented unreliable service.

[0071] RD: Connectionless (similar to UDP) reliable service.

[0072] RDMA (Remote Direct Memory Access) was created to address server-side data processing delays during network transmission. RDMA transfers data directly to a computer's storage area over the network, quickly moving data from one system to a remote system's memory without impacting the operating system. This reduces the need for computer processing power, eliminates the overhead of external memory copying and context switching, and frees up memory bandwidth and CPU cycles to improve application system performance.

[0073] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0074] FIG2 is a flowchart of an abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure.

[0075] Referring to FIG2 , the anomaly detection method of the RDMA system may include:

[0076] Step S202 : abstractly modeling the working space of the RDMA system to generate a search space of the RDMA system.

[0077] Step S204: determining the workload according to the dimension of the search space.

[0078] In an exemplary embodiment of the present disclosure, the workload may be computer devices such as servers, terminals, and intermediate nodes, but is not limited thereto.

[0079] Step S206: determining the conditional features of any workload triggering the RDMA system exception, so as to generate a minimum functional feature set of the exception.

[0080] In an exemplary embodiment of the present disclosure, the necessary conditions for triggering the anomaly are extracted by running a minimum feature set algorithm (MFS). The extracted feature set is used to avoid repeated searches of known anomaly areas, thereby accelerating the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary.

[0081] In an exemplary embodiment of the present disclosure, if an exception is triggered using an RC queue pair, a test is conducted to determine whether the exception can be reproduced using a UC queue. If the exception cannot be reproduced, it indicates that the RC queue is one of the MFS features of the exception, and the MFS outputs the necessary feature combination that triggers the exception.

[0082] Step S208: For any of the minimum functional feature sets, call the counter of the RDMA system to determine an abnormality indicator of the workload.

[0083] In an exemplary embodiment of the present disclosure, the counters include performance counters and / or diagnostic counters. The performance counters mainly reflect the performance of the RDMA subsystem, such as throughput, latency, CPU usage, etc., and the diagnostic counters mainly reflect errors or abnormal events within the RNIC, such as cache miss, internal congestion, etc., but are not limited to these.

[0084] The disclosed embodiments abstractly model the workspace of an RDMA system to generate a search space for the RDMA system. The workload is then determined based on the dimensions of the search space. The conditional features for triggering an RDMA system anomaly with any workload are determined to generate a minimum set of abnormal functional features. Finally, for any minimum set of functional features, the RDMA system's counters are called to determine workload anomaly indicators. This enables proactive detection of RDMA system performance anomalies and improves the performance and stability of the RDMA system in practical applications.

[0085] The following describes in detail the steps of the abnormality detection method for the RDMA system.

[0086] In an exemplary embodiment of the present disclosure, as shown in FIG3 , the present invention further includes:

[0087] Step S302: adjusting the workload pattern and / or the workload selection result based on the iterative calculation result of the abnormality indicator using a statistical algorithm.

[0088] In an exemplary embodiment of the present disclosure, as shown in FIG4 , adjusting the workload mode and / or the workload selection result based on the iterative calculation result of the abnormality indicator using a statistical algorithm includes:

[0089] Step S402 : calling the annealing algorithm in the statistical algorithm to iteratively calculate the count difference between the count value of the workload and the count value of the new workload.

[0090] Step S404: Determine a ratio between the count difference and the workload count value.

[0091] Step S406: Adjust the workload mode and / or the workload selection result according to the ratio value.

[0092] In an exemplary embodiment of the present disclosure, the workload pattern includes an access pattern, a message pattern, a request pattern, etc., but is not limited thereto. For example, a pattern in which RDMA access does not require confirmation from the remote machine at all, or a pattern in which RDMA access requires the participation of the remote machine's CPU. The workload pattern determines the generated RDMA traffic.

[0093] In an exemplary embodiment of the present disclosure, the workload selection result includes whether to move the workload causing the abnormality to a new workload.

[0094] In an exemplary embodiment of the present disclosure, as shown in FIG5 , adjusting the workload mode and / or the workload selection result according to the ratio value includes:

[0095] Step S502: If it is determined that the ratio value is less than zero, the workload is transferred to the new workload.

[0096] In an exemplary embodiment of the present disclosure, the above ratio is recorded as ΔE. If ΔE<0, the workload is transferred to the new workload.

[0097] Step S504: If it is determined that the proportional value is greater than zero, the inverse of the proportional value is calculated using an exponential function with the natural constant e as the base.

[0098] Step S506: Determine the calculation result of the exponential function as the probability of transferring the workload to the new workload.

[0099] In an exemplary embodiment of the present disclosure, if ΔE>0, exp(-ΔE / T) is calculated as the probability of transferring the workload to the new workload.

[0100] In an exemplary embodiment of the present disclosure, as shown in FIG6 , adjusting the workload mode and / or the workload selection result according to the ratio value includes:

[0101] Step S602: Adjust the request mode of the RDMA system and / or the allocation information of the buffer of the RDMA system according to the ratio value.

[0102] In an exemplary embodiment of the present disclosure, as shown in FIG7 , abstractly modeling the workspace of the RDMA system to generate the search space of the RDMA system includes:

[0103] Step S702: abstracting the workspace of the RDMA system to determine a memory area corresponding to the workspace, and registering the memory area.

[0104] Step S704: Create the queue pair and set the transmission type of the queue pair.

[0105] Step S706 : creating a work queue element according to the registered memory area and the queue pair with the transmission type set.

[0106] Step S708: Create the completion queue according to the work queue element to generate a search space of the RDMA system.

[0107] In an exemplary embodiment of the present disclosure, as shown in FIG8 , determining the workload according to the dimension of the search space includes:

[0108] Step S802: parsing the topology structure, host memory source, transmission mode and message mode included in the dimensions of the search space.

[0109] Step S804: Generate traffic between hosts of the RDMA system according to the topology, the host memory source, the transmission mode, and the message mode, and determine the workload corresponding to the traffic.

[0110] In an exemplary embodiment of the present disclosure, the transmission mode includes: QP type such as RC, UC, UD, etc.; the number of QPs; the operation code type; and the use of WQE.

[0111] The abnormality detection solution of the RDMA system disclosed in the present invention is described in detail below with reference to FIG9 to FIG11 .

[0112] In an exemplary embodiment of the present disclosure, as shown in FIG9 , a system architecture of an anomaly detection solution for an RDMA system 900 is provided. The RDMA system 900 includes a workload engine, an anomaly monitor, a workload generator, and the like.

[0113] In an exemplary embodiment of the present disclosure, the workload engine is responsible for establishing RDMA traffic.

[0114] In an exemplary embodiment of the present disclosure, anomaly monitoring detects performance bottlenecks and a minimum necessary anomaly feature set of MFS based on traffic and PFC pause frames.

[0115] In an exemplary embodiment of the present disclosure, the workload generator obtains indicators such as hardware count to determine the working mode of the test.

[0116] In an exemplary embodiment of the present disclosure, the above steps are repeated for multiple rounds, and finally a feature set of RDMA performance anomalies, that is, a minimum functional feature set, is obtained.

[0117] In an exemplary embodiment of the present disclosure, as shown in FIG10 , a search space 1000 for an anomaly detection solution for an RDMA system is provided. Based on the dimension parameters set in the search space, corresponding network traffic is generated between hosts through an RDMA network interface card. Specifically, the model is performed using the following four dimensions:

[0118] (1) Topology: How traffic flows into / out of the RNIC and to / from other server hard-commodity components.

[0119] (2) Memory allocation settings include configuring and dividing storage areas.

[0120] (3) Transmission settings: (a) QP type (RC, UC, UD); (b) Number of QPs; (c) Opcode type; (d) WQE usage. You can also set Opcodes such as SEND and WRITE.

[0121] (4) Message mode: You can flexibly set the message size and define the order of a series of messages.

[0122] Furthermore, the workload engine in the search space 1000 can combine and test different RDMA operations in the search space, abstractly model the workspace, and simulate the application's operations on RDMA, so as to quickly search for feature sets of RDMA performance anomalies.

[0123] In an exemplary embodiment of the present disclosure, as shown in FIG11 , an anomaly detection scheme for an RDMA system includes the following steps:

[0124] Step S1102: The workload setter constructs an abstract RDMA search space, including:

[0125] (1) Initialize S: workspace settings;

[0126] (2) Initialize temperature: annealing algorithm temperature indicator;

[0127] (3) Initialize N: workload dimension.

[0128] Step S1104: The workload engine generates RDMA traffic according to the workload pattern of the workload setter.

[0129] Step S1106: The abnormality monitor determines whether there is an abnormality based on the RDMA system indicators.

[0130] Step S1108: trigger an exception.

[0131] Step S1110: The anomaly monitor generates a minimum anomaly set using the MSF algorithm.

[0132] Step S1112: The workload setter obtains a minimum abnormal feature set.

[0133] The above steps S1104 to S1112 are iterated repeatedly, and finally a feature set of all detected RDMA performance anomalies is output.

[0134] Step S1114: The workload setter obtains the RDMA system counter.

[0135] Step S1116: The workload setter uses an annealing algorithm to calculate the energy function.

[0136] Step S1118: The workload setter operates an annealing algorithm to select whether to accept the workload based on the energy value and the annealing temperature.

[0137] Step S1120: The workload setter determines whether to update the annealing parameter temperature according to the number of iteration rounds.

[0138] Step S1122: The workload setter determines the next workload pattern according to the RDMA system (subsystem or main system) counter and the current search space.

[0139] Based on steps S1102 to S1122 above, a comprehensive search space is constructed from the developer's perspective by analyzing the standard verbs library and the design decisions that developers can make, such as request patterns and how to allocate RDMA buffers. Key abstractions in RDMA programming are constructed, including memory regions (MRs), queue pairs (QPs), work queues (WQEs), and completion queues (CQs).

[0140] Among these, we extracted four key dimensions that influence RDMA subsystem performance: host topology, memory allocation, transmission settings, and message mode. These abstractions have varying impacts on RDMA application performance under different settings. We simulated various workloads and used performance counters to identify anomalies.

[0141] The following is a detailed description of each modeling module:

[0142] (1) Memory Region (MR): The application must first register a memory region to make it accessible to RDMA. A function is used to register an MR. An MR has a starting address and length, defining a continuous block of memory. The RNIC can directly access the registered MR without CPU involvement.

[0143] (2) Queue Pair (QP): Use ibv_create_qp to create a QP. A QP represents a "connection" between the application and the RNIC. A QP is an abstraction for point-to-point communication and includes a send queue and a receive queue. Each QP needs to be configured with a transmission mode, such as reliable connection (RC).

[0144] (3) Work Queue Element (WQE): To send / receive messages, you need to construct a WQE and deliver it to the QP. The WQE contains a scatter / gather list that specifies a set of MR memory buffers that participate in the transmission. The WQE is delivered to the QP sending queue.

[0145] (3) Completion Queue (CQ): CQ is used to receive completion notifications and determine whether the WQE is completed. CQ is created and used to query CQ to obtain completion notifications. After obtaining the notification, the WQE memory buffer can be reused.

[0146] The specific workload generated according to the search space dimension can be modeled according to the following four dimensions:

[0147] (1) Topology: How traffic flows into / out of the RNIC and to / from other server hard-commodity components.

[0148] (2)Memory allocation settings.

[0149] (3) Transmission settings: (a) QP type (RC, UC, UD), you can set the Opcode such as SEND, WRITE, etc.; (b) The number of QPs; (c) Opcode type; (d) WQE usage.

[0150] (4) Message mode: The message size can be flexibly set, and the order of a series of messages can also be defined) Generate corresponding network traffic between hosts through the RDMA network interface card.

[0151] After the search space is constructed, the RDMA system's search process uses a simulated annealing algorithm to effectively search for application workloads that may trigger performance anomalies by driving the extreme values ​​of hardware counters.

[0152] The process of searching for RDMA performance anomalies using the annealing algorithm is as follows:

[0153] (1) Use the workload setter to select an initial random workload, i.e., the initial state.

[0154] (2) The workload engine generates RDMA traffic based on the workload pattern of the workload setter and sends it to the RDMA system.

[0155] (3) The anomaly detector performs anomaly detection based on the RDMA system anomaly indicators to determine whether an anomaly is triggered.

[0156] (4) If the anomaly detector detects a new anomaly, the anomaly monitor: runs the minimum feature set algorithm (MFS) to extract the necessary conditions to trigger the anomaly. The extracted feature set is used to avoid repeated searches of known anomaly areas, thereby speeding up the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary. For example: If an anomaly is triggered by an RC queue pair, test whether the anomaly can be reproduced using UC. If not, it indicates that RC is one of the MFS features of the anomaly, and MFS outputs the necessary feature combination to trigger the anomaly.

[0157] (5) The workload setter obtains the minimum anomaly feature set.

[0158] (6) The workload setter obtains RDMA system counters. It checks the counter values ​​at the workload point. It obtains counter interfaces through the RNIC, specifically performance counter values ​​and diagnostic counters. (These counter data do not rely on proprietary knowledge and do not require access to the internal implementation details of hardware such as the RNIC.)

[0159] (7) The workload setter uses an annealing algorithm to calculate the energy function. In this disclosure, the optimization goal is to make the performance counter as small as possible and the diagnostic counter as large as possible. The energy difference ΔE between the current point and the new point counter value is calculated. For the performance counter (to reduce its value as much as possible), ΔE is defined as: ΔE = (new counter value - old counter value) / old counter value. For the diagnostic counter (to increase its value as much as possible), ΔE is defined as: ΔE = (old counter value - new counter value) / new counter value. Performance counters mainly reflect the performance of the RDMA subsystem, such as throughput, latency, CPU usage, etc. The goal of this disclosure is to find workloads that cause performance anomalies (reduced throughput, increased latency, etc.). Therefore, if any workload makes the performance counter value smaller, it means that the performance is degraded and it is likely to trigger an anomaly. Therefore, for the performance counter, the smaller its value, the more likely it is to find a performance anomaly. Diagnostic counters mainly reflect errors or abnormal events within the RNIC, such as cache miss, internal congestion, etc. These counters usually only rise when there is a problem with the RNIC. Therefore, if any workload causes the diagnostic counter value to increase, it indicates that the RNIC internal anomaly has increased and there is a high probability of a problem. Therefore, for diagnostic counters, the larger the value, the more likely it is to find performance anomalies.

[0160] (8) The workload setter uses the annealing algorithm to calculate the energy function result and decide whether to accept it: if the energy change indicates that the counter value has been optimized, specifically ΔE<0, the new point is accepted; if the energy change indicates a deterioration, ΔE>0, the new workload is accepted with probability exp(-ΔE / T).

[0161] Furthermore, as the number of iterations increases, it is determined whether to change the temperature parameters of the annealing algorithm. As the number of search rounds increases, the probability acceptance metric is gradually reduced to make the search more focused.

[0162] (9) The workload setter determines whether to update the annealing temperature parameter based on the number of iterations.

[0163] (10) The workload setter determines the next workload pattern based on the RDMA subsystem counter and the current search space. The workload engine generates RDMA traffic based on the workload setter workload pattern and sends it to the RDMA system.

[0164] In summary, the above steps are repeated iteratively to finally output the feature set of all detected RDMA performance anomalies.

[0165] In an exemplary embodiment of the present disclosure, the embodiment of the present disclosure maintains a list of performance anomalies, each anomaly is a minimum fault region (corresponding to MFS), such as an area in the search space that causes the performance anomaly. The search starts with a random workload in the search space, and the algorithm of the embodiment of the present disclosure measures the value of a counter. In each iteration of SA, the embodiment of the present disclosure changes the workload in the search dimension. The embodiment of the present disclosure uses an anomaly monitor to test whether the new workload causes a performance anomaly. If so, the embodiment of the present disclosure runs the MFS algorithm of the embodiment of the present disclosure to determine the area in the entire search space that belongs to the anomaly. The embodiment of the present disclosure adds the new anomaly to the set and changes the current workload to a random workload. If the new workload does not trigger a performance anomaly, the embodiment of the present disclosure measures the point by comparing the value of the counter and decides whether to move the current workload to the new workload. For efficient search, the embodiment of the present disclosure always skips workloads that belong to existing performance anomalies.

[0166] Furthermore, the detection of abnormal conditions disclosed herein includes: in each round of iteration, the anomaly detector determines whether the new workload triggers an RDMA performance abnormality.

[0167] Abnormal detection condition 1: If pause frames are generated and the proportion of pause frames exceeds 0.1%, it is determined that an abnormal bottleneck is likely to occur.

[0168] Anomaly detection condition two: Each (remote network interface card) has its own specified maximum number of bits per second and maximum number of packets per second, which can be verified through simple benchmarking. If the workload's throughput is less than 20% of these limits, performance is likely limited by other bottlenecks in the RDMA (Remote Direct Memory Access) subsystem.

[0169] Furthermore, the process of extracting feature sets by the anomaly detector disclosed in the present invention is as follows:

[0170] If a new anomaly occurs, the anomaly monitor runs the minimum feature set algorithm (MFS) to extract the necessary conditions to trigger the anomaly. The extracted feature set avoids repeated searches of known anomaly areas, thereby speeding up the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary. For example: If an anomaly is triggered by an RC queue pair, test whether the anomaly can also be reproduced using UC. If not, it indicates that RC is one of the MFS features of the anomaly. MFS outputs the necessary feature combination that triggers the anomaly. After multiple rounds of repeated iterations, the feature set of all detected RDMA performance anomalies is finally output.

[0171] Corresponding to the above method embodiments, the present disclosure also provides an abnormality detection device for an RDMA system, which can be used to execute the above method embodiments.

[0172] FIG12 is a block diagram of an abnormality detection apparatus for an RDMA system in an exemplary embodiment of the present disclosure.

[0173] 12 , an abnormality detection apparatus 1200 of an RDMA system may include:

[0174] The modeling module 1202 is configured to perform abstract modeling on the working space of the RDMA system to generate a search space of the RDMA system.

[0175] The determination module 1204 is configured to determine the workload according to the dimension of the search space.

[0176] The determination module 1204 is configured to determine the conditional features of any workload triggering the RDMA system exception, so as to generate a minimum functional feature set of the exception.

[0177] The determining module 1204 is configured to call a counter of the RDMA system to determine an abnormality indicator of the workload for any of the minimum functional feature sets.

[0178] In an exemplary embodiment of the present disclosure, the abnormality detection apparatus 1200 of the RDMA system is further configured to:

[0179] The workload pattern and / or the workload selection result are adjusted based on the iterative calculation result of the abnormal indicator using a statistical algorithm.

[0180] In an exemplary embodiment of the present disclosure, the abnormality detection apparatus 1200 of the RDMA system is further configured to:

[0181] Calling an annealing algorithm in the statistical algorithm to iteratively calculate a count difference between the count value of the workload and the count value of a new workload;

[0182] determining a ratio between the count difference and the count value of the workload;

[0183] The workload mode and / or the workload selection result are adjusted according to the ratio value.

[0184] In an exemplary embodiment of the present disclosure, the abnormality detection apparatus 1200 of the RDMA system is further configured to:

[0185] If it is determined that the ratio value is less than zero, transferring the workload to the new workload;

[0186] If it is determined that the proportional value is greater than zero, calculating the inverse of the proportional value using an exponential function with the natural constant e as the base;

[0187] A calculation result of the exponential function is determined as a probability of transferring the workload to the new workload.

[0188] In an exemplary embodiment of the present disclosure, the abnormality detection apparatus 1200 of the RDMA system is further configured to:

[0189] The request mode of the RDMA system and / or allocation information of the buffer of the RDMA system are adjusted according to the ratio value.

[0190] In an exemplary embodiment of the present disclosure, the modeling module 1202 is further configured to:

[0191] Abstracting the workspace of the RDMA system to determine a memory area corresponding to the workspace, and registering the memory area;

[0192] Creating the queue pair and setting the transmission type of the queue pair;

[0193] Create a work queue element based on the registered memory area and queue pair with the transfer type set;

[0194] The completion queue is created according to the work queue elements to generate a search space of the RDMA system.

[0195] In an exemplary embodiment of the present disclosure, the determining module 1204 is further configured to:

[0196] Parsing the dimensions of the search space including topology, host memory resources, transmission mode, and message mode;

[0197] Traffic between hosts of the RDMA system is generated according to the topology, the host memory source, the transmission mode, and the message mode, and the workload corresponding to the traffic is determined.

[0198] Since the functions of the apparatus 1200 have been described in detail in the corresponding method embodiments, they will not be described in detail in this disclosure.

[0199] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the conditional features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the conditional features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0200] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0201] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0202] The electronic device 1300 according to this embodiment of the present invention is described below with reference to Figure 13. The electronic device 1300 shown in Figure 13 is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0203] As shown in Figure 13, electronic device 1300 is implemented as a general-purpose computing device. Components of electronic device 1300 may include, but are not limited to, the aforementioned at least one processing unit 1310, the aforementioned at least one storage unit 1320, and a bus 1330 connecting various system components (including storage unit 1320 and processing unit 1310).

[0204] The storage unit stores program code that can be executed by the processing unit 1310, causing the processing unit 1310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1310 can perform the method described in the embodiments of the present disclosure.

[0205] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 13201 and / or a cache memory unit 13202 , and may further include a read-only memory unit (ROM) 13203 .

[0206] The storage unit 1320 may also include a program / utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0207] Bus 1330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0208] Electronic device 1300 may also communicate with one or more external devices 1340 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1300, and / or any device that enables electronic device 1300 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication may occur via input / output (I / O) interface 1350. Furthermore, electronic device 1300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1360. As shown, network adapter 1360 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with electronic device 1300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0209] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0210] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0211] The program product for implementing the above-described method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0212] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0213] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0214] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0215] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0216] Furthermore, the figures above are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0217] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims. Industrial Applicability

[0218] The disclosed embodiments abstractly model the workspace of an RDMA system to generate a search space for the RDMA system. The workload is then determined based on the dimensions of the search space. The conditional features for triggering an RDMA system anomaly with any workload are determined to generate a minimum set of abnormal functional features. Finally, for any minimum set of functional features, the RDMA system's counters are called to determine workload anomaly indicators. This enables proactive detection of RDMA system performance anomalies and improves the performance and stability of the RDMA system in practical applications.

Claims

1. An abnormal detection method for an RDMA system, characterized in that it includes: Abstractly model the working space of the RDMA system to generate a search space for the RDMA system; Determine the workload according to the dimension of the search space; Determine the conditional features for any of the workloads to trigger an abnormality in the RDMA system to generate a minimum functional feature set for the abnormality; For any of the minimum functional feature sets, call the counter of the RDMA system to determine the abnormality index of the workload.

2. The abnormal detection method for an RDMA system according to claim 1, characterized in that it further includes: Adjust the mode of the workload and / or the selection result of the workload based on the iterative calculation result of the abnormality index by a statistical algorithm.

3. The anomaly detection method for the RDMA system according to claim 2, wherein the conditional feature is that, Adjusting the mode of the workload and / or the selection result of the workload based on the iterative calculation result of the abnormality index by a statistical algorithm includes: Call the annealing algorithm in the statistical algorithm to iteratively calculate the count difference between the count value of the workload and the count value of the new workload; Determine the ratio value between the count difference and the count value of the workload; Adjust the mode of the workload and / or the selection result of the workload according to the ratio value.

4. The anomaly detection method of the RDMA system according to claim 3, wherein the conditional feature is that, Adjusting the mode of the workload and / or the selection result of the workload according to the ratio value includes: If it is determined that the ratio value is less than zero, transfer the workload to the new workload; If it is determined that the ratio value is greater than zero, calculate the opposite number of the ratio value with the exponential function with the natural constant e as the base; Determine the calculation result of the exponential function as the probability of transferring the workload to the new workload.

5. The abnormal detection method for an RDMA system according to claim 3, characterized in that adjusting the mode of the workload and / or the selection result of the workload according to the ratio value includes: Adjust the request mode of the RDMA system and / or the allocation information of the buffer of the RDMA system according to the ratio value.

6. The anomaly detection method for the RDMA system according to any one of claims 1-5, wherein Abstractly modeling the working space of the RDMA system to generate a search space for the RDMA system includes: Abstract the working space of the RDMA system to determine the memory area corresponding to the working space, and register the memory area; Create the queue pair and set the transmission type of the queue pair; Create a work queue element according to the registered memory area and the queue pair with the transmission type set; Create the completion queue according to the work queue element to generate a search space for the RDMA system.

7. The abnormal detection method of the RDMA system according to any one of claims 1-5, wherein the condition is characterized in that, Determining the workload according to the dimension of the search space includes: Parse the topology structure, host memory source, transmission mode, and message mode included in the dimension of the search space; Generate the traffic between the hosts of the RDMA system according to the topology structure, the host memory source, the transmission mode, and the message mode, and determine the workload corresponding to the traffic.

8. An abnormal detection device for an RDMA system, characterized in that it includes: A modeling module, configured to abstractly model the working space of the RDMA system to generate a search space of the RDMA system; A determination module, configured to determine a workload according to the dimension of the search space; The determination module is configured to determine the conditional characteristics for any one of the workloads to trigger an exception in the RDMA system, so as to generate a minimum functional characteristic set of the exception; The determination module is configured to, for any one of the minimum functional characteristic sets, call a counter of the RDMA system to determine an exception index of the workload.

9. An electronic device, characterized in that it includes: A memory; And A processor coupled to the memory, the processor being configured to execute the exception detection method of the RDMA system according to any one of claims 1-7 based on instructions stored in the memory.

10. A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the exception detection method of the RDMA system according to any one of claims 1-7 is implemented.

Images