Layered monitoring method and system for RDMA (Remote Direct Memory Access) memory decoupling system
By employing a hierarchical monitoring approach in the RDMA memory decoupling system, and combining passive monitoring with active probing and health assessment, the problems of centralized dependency and fine-grained detection are solved. This enables efficient and accurate health status assessment of memory nodes, thereby improving the reliability and performance of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies in RDMA memory decoupling systems suffer from centralized dependence, poor adaptability of passive nodes, and inability to achieve fine-grained detection, resulting in insufficient timeliness and accuracy of fault detection, resource waste, and over-recovery.
A hierarchical monitoring approach is adopted, which performs passive monitoring, active probing and health assessment steps in computing nodes, calculates error rate and access latency using a sliding window, and combines health table and health risk score to achieve fine-grained monitoring and health status assessment of queue pairs (QP).
This improves the reliability of the RDMA memory decoupling system, enables lightweight monitoring and accurate health status assessment of memory nodes, reduces misjudgments and resource waste, and ensures high system reliability and performance.
Smart Images
Figure CN121807651A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data center memory reliability optimization, and more specifically, relates to a hierarchical monitoring method and system for RDMA memory decoupling systems. Background Technology
[0002] Modern data centers are widely adopting disaggregated memory (DM) architecture, which physically separates compute nodes from memory nodes and enables high-speed communication through remote direct memory access (RDMA) to improve resource utilization, flexibility, and scalability. However, in a disaggregated memory system, memory becomes a remote resource that needs to be accessed via a network, which places stringent requirements on the timeliness and accuracy of fault detection.
[0003] Traditional distributed systems often rely on lease-based mechanisms for fault detection, where each node periodically renews its lease with a centralized master node, and failure to renew on time is considered a fault. However, this mechanism has significant drawbacks in RDMA-based memory decoupling systems: First, centralized lease management suffers from scalability bottlenecks, with the master node's load increasing linearly with the number of nodes, and it is prone to becoming a single point of failure, triggering a "recovery storm." Second, memory nodes in memory decoupling systems are often "pure memory pools" without CPUs or operating systems, unable to proactively renew leases, thus breaking the core assumption of the lease mechanism. Third, RDMA communication lacks retransmission and automatic recovery mechanisms; lease data packets may be silently discarded due to credit exhaustion, QP errors, etc., leading to misjudgments of healthy nodes. Fourth, the detection granularity is coarse, only enabling node-level detection and failing to identify fine-grained partial faults such as memory blocks, links, and QPs, resulting in resource waste and over-recovery.
[0004] In existing technologies, some solutions attempt to improve reliability by optimizing lease timeout thresholds or increasing QP redundancy, but they fail to address core issues such as centralized dependency, passive node adaptability, and fine-grained detection. Therefore, there is an urgent need to design a health monitoring method adapted to the characteristics of RDMA memory decoupling systems to overcome the limitations of traditional mechanisms. Summary of the Invention
[0005] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a hierarchical monitoring method and system for RDMA memory decoupling systems, the purpose of which is to improve the reliability of memory decoupling systems.
[0006] To achieve the above objectives, according to one aspect of the present invention, a hierarchical monitoring method for an RDMA memory decoupling system is provided, including a passive monitoring step, an active probing step, and a health assessment step performed in a computing node; The passive monitoring steps include: real-time polling of entries in the RDMA completion queue and writing them to the local circular buffer of the compute node; using a sliding window to extract the latest partial entries in the circular buffer, calculating the error rate and sample latency of the monitoring unit based on the entries in the sliding window, and updating the health table; the monitoring unit is a queue pair; The active detection steps include: sending RDMA detection packets to each monitoring unit at a preset detection frequency, or immediately performing detection on suspicious units marked by passive monitoring, and updating the health table based on the detection results; the detection results will be used as auxiliary observation information and comprehensively analyzed together with the passive monitoring results; The health assessment steps include: obtaining the latest health information of each monitoring unit from the health table according to the preset assessment frequency, calculating the health risk score of each monitoring unit, and then updating the health table; The health table records the health information of each monitoring unit. The health information includes error rate, access latency, probe latency, number of consecutive probe failures, and health risk score. The health risk score is calculated based on the error rate, access latency, and probe latency. The RDMA probe packet is an 8-64 byte RDMA request packet.
[0007] Furthermore, the passive monitoring steps also include: If the error rate of a monitoring unit exceeds a preset first threshold, or if the difference between the access latency and the preset baseline latency exceeds a preset second threshold, the monitoring unit will be marked as a suspicious unit.
[0008] Furthermore, the health assessment steps also include: if the monitoring unit's health risk score... Then mark the monitoring unit as a healthy unit; if Then the monitoring unit will be marked as a suspicious unit; if If so, the monitoring unit will be marked as a faulty unit; in, and These represent the minimum and maximum values of the health risk score, respectively. .
[0009] Furthermore, the active detection steps also include: When a monitoring unit is marked as a suspicious unit, RDMA probe packets are continuously sent to the suspicious unit, and the health table is updated based on the probe results. The probe results will be used as auxiliary observation information and comprehensively analyzed together with the passive monitoring results until the health risk score of the suspicious unit falls into the range. or ; in, and These represent the minimum and maximum values of the health risk score, respectively. .
[0010] Furthermore, the calculation method for the health risk score of the monitoring unit includes:
[0011]
[0012] in, Indicates the current moment; This indicates the error rate of the monitoring unit at the current moment. This represents the difference between the current access latency of the monitoring unit and the preset baseline latency. This represents the difference between the detection delay of the monitoring unit at the current moment and the baseline delay. , and They represent , and The weights; Indicates the current time , and The weighted summation result, This indicates the health risk score of the monitoring unit at the current moment. This indicates the health risk score of the monitoring unit at the previous moment; This represents the preset smoothing factor.
[0013] Furthermore,
[0014] in, This indicates the number of failed operations corresponding to the monitoring unit within the sliding window at the current moment. This indicates the total number of requests corresponding to the monitoring units within the sliding window at the current moment.
[0015] Furthermore,
[0016] in, This indicates the latest access delay at the current moment. This indicates a delay in access from the previous moment. This represents the access latency of each monitoring unit at the current moment. This is the preset smoothing factor.
[0017] Furthermore, the hierarchical monitoring method for the RDMA memory decoupling system provided by this invention further includes the following health assessment step: If the number of consecutive detection failures of the monitoring unit exceeds the preset third threshold, the monitoring unit will be marked as a faulty unit. If the difference between the detection delay of the monitoring unit and the preset baseline delay is continuous If the link exceeds the preset fourth threshold once, the link corresponding to the monitoring unit will be marked as an abnormal link. in, It is a positive integer.
[0018] Furthermore, the health assessment step also includes: aggregating the health information of monitoring units belonging to the same memory node to obtain the health information of the memory node.
[0019] According to another aspect of the present invention, a hierarchical monitoring system for an RDMA memory decoupling system is provided, deployed on each computing node, the hierarchical monitoring system comprising: A computer-readable storage medium for storing computer programs; And a processor for reading computer programs stored in a computer-readable storage medium to implement the hierarchical monitoring method for the RDMA memory decoupling system provided by this invention.
[0020] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention uses queue pairs (QP) as monitoring units, which can realize fine-grained monitoring. This invention polls the entries in the RDMA completion queue (CQ) in real time on the computing node, and calculates the error rate and access latency of the monitoring unit (i.e., queue pair). It realizes continuous passive monitoring of memory nodes without additional communication overhead. By sending RDMA probe requests with small resource consumption, it performs auxiliary observation of abnormal monitoring units without significantly affecting normal data access, so as to count the probe latency and the number of consecutive probe failures of the monitoring units, and realizes lightweight active probing of memory nodes. In addition, this invention periodically integrates the results of passive monitoring and active probing to calculate the health risk score of the monitoring unit. This enables accurate assessment of the health status of the monitoring unit without relying on the master node, and improves the reliability of the RDMA memory decoupling system.
[0021] (2) In the preferred embodiment of the present invention, during the passive monitoring process, the monitored objects with high error rates and access delays that significantly deviate from the baseline delay are marked as suspicious units; during the health assessment process, the monitored units are marked as healthy units, suspicious units and faulty units according to the range of the health risk scores of the monitored units, so as to provide accurate basis for the upper-level recovery decision.
[0022] (3) In a preferred embodiment of the present invention, a detection process is immediately performed on a suspicious unit until the state of the suspicious unit becomes a healthy unit or a faulty unit. This ensures high system performance by avoiding unnecessary recovery operations on healthy units while ensuring timely recovery of faulty units to guarantee high reliability.
[0023] (4) In a preferred embodiment of the present invention, the present invention uses a smoothing function to update the health risk score, thereby avoiding drastic fluctuations in the score. In a further preferred embodiment, the calculated access delay is also updated using a smoothing function, thereby balancing recent anomalies with historical stable states and avoiding mislabeling caused by transient jitter.
[0024] (5) In a preferred embodiment of the present invention, the faulty unit and abnormal link are further identified based on the number of consecutive detection failures and detection delay in the active detection results, thereby providing richer and more accurate basis for upper-level decision-making and further improving system reliability. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of a hierarchical monitoring method for an RDMA memory decoupling system provided in an embodiment of the present invention.
[0026] Figure 2 A flowchart of a hierarchical monitoring method for an RDMA memory decoupling system provided in an embodiment of the present invention.
[0027] Figure 3 This is a schematic diagram illustrating the collaboration between event-driven and lightweight detection, as provided in an embodiment of the present invention. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0029] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0030] To overcome the core problems of traditional health monitoring mechanisms, such as centralized dependence, poor adaptability of passive nodes, and inability to achieve fine-grained detection, this invention provides a hierarchical monitoring method and system for RDMA memory decoupling systems. The overall concept is to use queue pairs (QPs) as monitoring units. Based on this, through the collaborative work of passive monitoring, lightweight detection, and health scoring, accurate health monitoring is achieved, providing precise basis for upper-level recovery decisions, thereby effectively improving the reliability of RDMA memory decoupling systems.
[0031] Before explaining the technical solution of the present invention in detail, the relevant technical terms are briefly introduced as follows: RDMA: Remote Direct Memory Access is a direct memory access technology that transfers data directly from the memory of one computer to another without the intervention of the operating systems of both computers. This technology allows for high-throughput, low-latency network communication and is particularly suitable for use in large-scale parallel computer clusters.
[0032] The RDMA Completion Queue (CQ) is a hardware data structure used by the network interface card (NIC) to notify the application of the completion status of an operation. After each RDMA operation is executed, the NIC generates a record in the completion queue, containing information such as whether the operation was successful, the duration, and the amount of data transferred. The information recorded in the RDMA completion queue entry includes the operation type, status code, number of bytes transferred, queue pair (QP) identifier, and timestamp. The operation type includes three categories: remote read, remote write, and message sending. The status code indicates the result of the operation, whether it was successful or failed, and the specific reason for the failure (such as network transmission error, remote node not responding, connection abnormality, etc.).
[0033] A queue pair (QP) is the basic channel unit for RDMA communication. Each pair of nodes needs to establish a dedicated queue pair to communicate. A queue pair contains a transmit queue and a receive queue, and is responsible for managing all data transmission on that channel.
[0034] A computing node (CN) is the main entity that executes applications and data processing, and is typically responsible for initiating RDMA operations (such as read or write).
[0035] A memory node (MN) is responsible for managing shared memory resources and providing a global buffer pool accessible to multiple compute nodes.
[0036] The RDMA memory decoupling system involved in this invention is a memory decoupling cluster based on RDMA support, and the cluster includes compute nodes, memory nodes and switches.
[0037] The following is an example.
[0038] Example 1: A hierarchical monitoring method for an RDMA memory decoupling system includes passive monitoring steps, active probing steps, and health assessment steps performed in the compute nodes. Figure 1 , Figure 2 and Figure 3 As shown, in this embodiment, the passive monitoring steps include: polling the entries in the RDMA completion queue in real time and writing them into the local circular buffer of the computing node; using a sliding window to extract the latest partial entries in the circular buffer, calculating the error rate and sample latency of the monitoring unit based on the entries in the sliding window, and updating the health table; the monitoring unit is a queue pair; The active detection steps include: sending RDMA detection packets to each monitoring unit at a preset detection frequency, or immediately performing detection on suspicious units marked by passive monitoring, and updating the health table based on the detection results; the detection results will be used as auxiliary observation information and comprehensively analyzed together with the passive monitoring results; The health assessment steps include: obtaining the latest health information of each monitoring unit from the health table according to the preset assessment frequency, calculating the health risk score of each monitoring unit, and then updating the health table; The health table records the health information of each monitoring unit. The health information includes error rate, access latency, probe latency, number of consecutive probe failures, and health risk score. The health risk score is calculated based on the error rate, access latency, and probe latency. The RDMA probe packet is an 8-64 byte RDMA request packet.
[0039] Optionally, in this embodiment, the health table is a lock-free hash table, where each entry corresponds to a monitoring unit (i.e., QP). The health table is implemented based on the lock-free hash table, and the concurrent safe data structure supports multi-threaded simultaneous reading and writing, avoiding the overhead of traditional locks.
[0040] This embodiment uses queue pairs (QPs) as monitoring units, enabling fine-grained monitoring. RDMA completion queue entries inherently carry QP numbers (qp_num), making it easiest to "account by QP" for error rate and access latency statistics. This embodiment polls entries in the RDMA completion queue (CQ) in real time on the compute node, calculating the error rate and access latency of the monitoring unit (i.e., queue pair). This achieves continuous passive monitoring of memory nodes without additional communication overhead. By sending RDMA probe requests with low resource consumption (8-64 bytes), auxiliary observation of abnormal monitoring units is performed without significantly affecting normal data access, to statistically analyze the probe latency and consecutive probe failures of the monitoring units, achieving lightweight active probing of memory nodes. Furthermore, this embodiment periodically integrates the results of passive monitoring and active probing to calculate a health risk score for the monitoring unit. This allows for accurate assessment of the health status of the monitoring unit without relying on the master node, improving the reliability of the RDMA memory decoupling system.
[0041] In this embodiment, passive monitoring improves concurrent processing efficiency through a circular buffer and a lock-free hash table; lightweight probing avoids traffic surges through fixed-rate scheduling; and health risk scoring uses simple arithmetic operations to ensure negligible computational overhead.
[0042] Optionally, in this embodiment, the health risk score ranges from 0 to 1, and within this range, a higher score indicates a greater risk of failure. The specific formula for calculating the health risk score is as follows:
[0043]
[0044] in, Indicates the current moment; This indicates the error rate of the monitoring unit at the current moment. This represents the difference between the current access latency of the monitoring unit and the preset baseline latency. This represents the difference between the detection delay of the monitoring unit at the current moment and the baseline delay. , and They represent , and The weights; Indicates the current time , and The weighted summation result, This indicates the health risk score of the monitoring unit at the current moment. This indicates the health risk score of the monitoring unit at the previous moment; This represents the preset smoothing factor.
[0045] It reflects clear operational failures (such as timeouts and QP errors) and quickly responds to hardware failures such as network card disconnection and memory damage. It reflects the degree to which latency deviates from the baseline and captures soft degradations such as link congestion and slower access. Active detection uncovers cold data anomalies and silent packet loss in RDMA data sources that are not covered by passive monitoring. These three indicators provide complementary signals across different traffic scenarios, fault modes, and time scales, avoiding blind spots and false alarms associated with single indicators. Finally, a weighted fusion is performed to generate a health risk score, achieving a more robust and comprehensive health assessment. Optionally, in this embodiment, , , This weight setting can be adapted to general workloads; .
[0046] This embodiment, based on the calculation of health risk scores through weighted summation, further employs a smoothing function to fuse the calculation results at the current moment with the health risk scores at the previous moment, thereby avoiding drastic fluctuations in scores.
[0047] Based on health risk scoring, this embodiment classifies the monitoring unit into three levels: healthy, suspicious, and faulty. Specifically, the active detection steps in this embodiment further include: The health assessment process also includes: if the monitoring unit's health risk score... Then mark the monitoring unit as a healthy unit; if Then the monitoring unit will be marked as a suspicious unit; if If so, the monitoring unit will be marked as a faulty unit; in, and These represent the minimum and maximum values of the health risk score, respectively. Optionally, in this embodiment, , , , .
[0048] In the passive monitoring step of this embodiment, based on real-time polling of RDMA completion queue entries, a sliding window is used to limit the time range of statistical indicators, retaining only the data from the most recent T seconds or the most recent N operations. This allows health monitoring to focus on the "current state" without being diluted by historical normal period data, filtering out premature transient jitters and avoiding their long-term impact on the score.
[0049] In this implementation, The calculation method is as follows:
[0050] in, This indicates the number of failed operations corresponding to the monitoring unit within the sliding window at the current moment. This represents the total number of requests corresponding to the monitoring units within the sliding window at the current moment. In this embodiment, the error rate is calculated as the ratio of the number of failed operations within the sliding window to the total number of requests, which can intuitively reflect the failure probability of the unit.
[0051] In this embodiment, the access latency is calculated using the Exponentially Weighted Moving Average (EWMA). Therefore, the access latency in this embodiment is an EWMA latency, and its calculation method is as follows:
[0052] in, This indicates the latest access delay at the current moment. This indicates a delay in access from the previous moment. This represents the access latency of each monitoring unit at the current moment. This is a preset smoothing factor. In this embodiment, the calculated access latency is also updated using a smoothing function, which can balance recent anomalies with historical stable states and avoid mislabeling caused by transient jitter. Optionally, in this embodiment, the smoothing factor... To balance sensitivity and stability.
[0053] The passive monitoring steps in this embodiment also include: If the error rate of a monitoring unit exceeds a preset first threshold, or the difference between the access latency and a preset baseline latency exceeds a preset second threshold, the monitoring unit is marked as a suspicious unit. Optionally, in this embodiment, the first threshold is 5%, and the second threshold is 30%.
[0054] For suspicious units, if they are actually faulty, failure to perform recovery operations in a timely manner may severely impact system reliability; if they are actually healthy, performing recovery operations on them may severely impact system performance. To ensure system reliability without affecting system performance, this embodiment immediately initiates an observation period for the detected suspicious unit, during which active probing is continuously performed. Accordingly, the active probing steps in this embodiment further include: When a monitoring unit is marked as a suspicious unit, RDMA probe packets are continuously sent to the suspicious unit, and the health table is updated based on the probe results. The probe results will be used as auxiliary observation information and comprehensively analyzed together with the passive monitoring results until the health risk score of the suspicious unit falls into the range of 100%. or This means that a cell can be remarked as a healthy cell or a faulty cell.
[0055] This implementation employs a "periodic sampling + on-demand triggering" strategy for active detection. Periodic sampling covers cold memory units (i.e., monitoring units not accessed by upper-layer applications), while on-demand triggering promptly verifies the actual status of these units. Evaluation results are assessed through continuous failure counting, latency deviation, and status code tracking, filling blind spots in passive monitoring and reducing false alarms. Specifically, in this embodiment, the active detection frequency is 60 seconds.
[0056] Based on the health monitoring results of QPs, node-level health monitoring results can be aggregated. The group of QPs "connected to the same remote memory node" (and the memory block statistics accessed by these QPs if necessary) are grouped together for statistics: if only a single QP is abnormal, a link / channel level problem occurs; if a group of QPs connected to the same remote memory node meets the preset fault conditions, such as continuously being abnormal in the same time window (high error rate, simultaneous spike in latency, and failure of probes), it is determined that the remote memory node (MN) has experienced a node-level fault.
[0057] Based on the above analysis, in this embodiment, the health assessment step further includes: aggregating the health information of monitoring units belonging to the same memory node to obtain the health information of the memory node.
[0058] As a preferred embodiment, the health assessment step in this example further includes: If the number of consecutive detection failures of the monitoring unit exceeds the preset third threshold, the monitoring unit will be marked as a faulty unit. If the difference between the detection delay of the monitoring unit and the preset baseline delay is continuous If the link exceeds the preset fourth threshold once, the link corresponding to the monitoring unit will be marked as an abnormal link. in, It is a positive integer; optionally, in this embodiment, .
[0059] The background thread evaluates the probe results every 50 milliseconds.
[0060] In practical applications, the health score of each monitoring unit is initialized to 0.1. Passive monitoring and detection data are aggregated every 100 milliseconds, the health risk score of each monitoring unit is recalculated, and a smoothing function is used to update the health risk score. The status of the monitoring unit is marked based on the updated health risk score. If it is marked as faulty, risk warning information is provided to the upper-layer system, and the upper-layer policy determines whether to perform recovery operations (such as memory block migration, QP reconstruction). When the status of a monitoring unit is suspicious, the priority of detection resource allocation is increased by 2 times to accelerate status confirmation. After the status is restored to healthy, the priority is restored to the default value.
[0061] The health assessment step in this embodiment further involves periodically calculating the health risk score distribution of each monitoring unit. Specifically, the health risk score distribution of each monitoring unit is calculated every 500 milliseconds. Units with a score ≥ 0.5 are classified as high-risk and allocated 60% of the total detection resources; units with a score between 0.3 and 0.5 are classified as medium-risk and allocated 30% of the resources; and the remaining low-risk units are allocated 10% of the resources. Based on this, resource allocation is achieved by dynamically adjusting the execution frequency of the detection tasks: high-risk units are detected once every 10 seconds, medium-risk units are detected once every 30 seconds, and low-risk units maintain a periodic sampling of once every 60 seconds.
[0062] Optionally, the deployment environment of the memory decoupling system involved in this embodiment is as follows: The compute nodes are equipped with network interface cards (NICs) supporting RDMA (such as the Mellanox ConnectX series), and the memory nodes are pure memory pools without CPUs, achieving low-latency communication through networks with speeds of 56Gbps and above. In practical implementation, at the software level, this embodiment deploys a passive monitoring module at the network driver layer of the compute nodes, and a lightweight detection module and a health scoring module at the application layer, ensuring compatibility with existing memory decoupling systems (such as Clover). The passive monitoring module, lightweight detection module, and health scoring module are used to implement the aforementioned passive monitoring steps, active detection steps, and health assessment steps, respectively.
[0063] Example 2: A hierarchical monitoring system for an RDMA memory decoupling system is deployed on each computing node. The hierarchical monitoring system includes: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in a computer-readable storage medium to implement the hierarchical monitoring method for the RDMA memory decoupling system provided in Embodiment 1 above.
[0064] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A hierarchical monitoring method for an RDMA memory decoupling system, characterized in that, This includes passive monitoring steps, active detection steps, and health assessment steps performed in the computing nodes; The passive monitoring steps include: real-time polling of entries in the RDMA completion queue and writing them to the local circular buffer of the compute node; using a sliding window to extract the latest partial entries in the circular buffer, and calculating the error rate and access latency of the monitoring unit based on the entries in the sliding window, and updating the health table; the monitoring unit is a queue pair; The active detection step includes: sending RDMA detection packets to each monitoring unit at a preset detection frequency, or immediately performing detection on suspicious units marked by passive monitoring, and updating the health table based on the detection results; The health assessment steps include: obtaining the latest health information of each monitoring unit from the health table according to a preset assessment frequency, calculating the health risk score of each monitoring unit, and then updating the health table; The health table is used to record the health information of each monitoring unit; the health information includes error rate, access latency, probe latency, number of consecutive probe failures and health risk score; the health risk score is calculated based on the error rate, access latency and probe latency; the RDMA probe packet is an 8-64 byte RDMA request packet.
2. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 1, characterized in that, The passive monitoring steps also include: If the error rate of a monitoring unit exceeds a preset first threshold, or if the difference between the access latency and the preset baseline latency exceeds a preset second threshold, the monitoring unit will be marked as a suspicious unit.
3. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 1, characterized in that, The health assessment step also includes: if the monitoring unit has a health risk score Then mark the monitoring unit as a healthy unit; if Then the monitoring unit will be marked as a suspicious unit; if If so, the monitoring unit will be marked as a faulty unit; in, and These represent the minimum and maximum values of the health risk score, respectively. .
4. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 2 or 3, characterized in that, The active detection step also includes: When a monitoring unit is marked as a suspicious unit, RDMA probe packets are continuously sent to the suspicious unit, and the health table is updated based on the probe results until the health risk score of the suspicious unit falls into the range. or ; in, and These represent the minimum and maximum values of the health risk score, respectively. .
5. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 1, characterized in that, The calculation methods for the health risk score of the monitoring unit include: in, Indicates the current moment; This indicates the error rate of the monitoring unit at the current moment. This represents the difference between the current access latency of the monitoring unit and the preset baseline latency. This represents the difference between the detection delay of the monitoring unit at the current moment and the baseline delay. , and They represent , and The weights; Indicates the current time , and The weighted summation result, This indicates the health risk score of the monitoring unit at the current moment. This indicates the health risk score of the monitoring unit at the previous moment; This represents the preset smoothing factor.
6. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 5, characterized in that, in, This indicates the number of failed operations corresponding to the monitoring unit within the sliding window at the current moment. This indicates the total number of requests corresponding to the monitoring units within the sliding window at the current moment.
7. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 5, characterized in that, in, This indicates the latest access delay at the current moment. This indicates a delay in access from the previous moment. This represents the access latency of each monitoring unit at the current moment. This is the preset smoothing factor.
8. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 1, characterized in that, The health assessment steps also include: If the number of consecutive detection failures of the monitoring unit exceeds the preset third threshold, the monitoring unit will be marked as a faulty unit. If the difference between the detection delay of the monitoring unit and the preset baseline delay is continuous If the link exceeds the preset fourth threshold once, the link corresponding to the monitoring unit will be marked as an abnormal link. in, It is a positive integer.
9. The hierarchical monitoring method for an RDMA memory decoupling system as described in claim 1, characterized in that, The health assessment step further includes: aggregating the health information of monitoring units belonging to the same memory node to obtain the health information of the memory node.
10. A hierarchical monitoring system for an RDMA memory decoupling system, deployed on each computing node, characterized in that, The hierarchical monitoring system includes: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium to implement the hierarchical monitoring method for the RDMA memory decoupling system according to any one of claims 1 to 9.