RDMA / TCP Dual Transmission Plane Cooperative QoS Scheduling Method and System

Abstract

This application provides an RDMA / TCP dual-transmission plane collaborative QoS scheduling method and system, including establishing Remote Direct Memory Access (RDA) read / write token buckets and Transmission Control Protocol (TCP) read / write token buckets for the same tenant at the block device layer; configuring a first time slice granularity and a first smoothing suppression strength for the RDA plane, and configuring a second time slice granularity and a second smoothing suppression strength for the TCP plane; collecting the flow control trigger count, explicit congestion notification marking count, and retries count of the previous plane, and the round-trip delay, retransmission count, and congestion window size of the subsequent plane, and adjusting the filling rate and suppression parameters of the corresponding token buckets accordingly; when any plane fails, newly arriving IO requests are bound to the token bucket scheduling of the healthy plane, and in-process IO requests that have been admitted through the failed plane and issued before the failure continue to execute. This application implements protocol-differentiated rate control, network state adaptive adjustment, and transparent fault migration.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to an RDMA / TCP dual-transmission plane collaborative QoS scheduling method and system. Background Technology

[0002] In large-scale AI model training and inference scenarios, storage networks typically handle both Remote Direct Memory Access (RDMA) and Transmission Control Protocol (TCP) traffic. RDMA is used for high-speed data loading, while TCP is used for management communication and cross-domain backup. Existing block device layer Quality of Service (QoS) mechanisms maintain only a single token bucket. Read and write requests from both protocols consume the same token limit, and RDMA's large-granularity sequential requests are frequently interrupted by TCP's small packets, significantly reducing effective throughput. Furthermore, when network congestion occurs, TCP shrinks its window through congestion control, and RDMA actively slows down through link-layer flow control. However, the block device layer's token bucket continues to fill at a fixed rate, resulting in a double suppression of network speed reduction coupled with continued block-layer rate limiting, reducing throughput to an extremely low proportion of physical bandwidth. Moreover, if the RDMA plane becomes unavailable due to network card failure or switch congestion, existing solutions rely on application-layer timeout retries to switch to TCP, resulting in switching latency on the order of seconds. This causes complete blocking of checkpoint writes during AI training, leaving the GPU idle and waiting. The above problems stem from the lack of an information path between the block layer service quality and the actual state of the transport layer, and the fact that the differentiated transmission characteristics of the two protocols are not reflected in the block layer admission control, and the consistency and bandwidth continuity of ongoing I / O cannot be guaranteed during fault switching. Summary of the Invention

[0003] This application aims to provide an RDMA / TCP dual-transmission plane collaborative QoS scheduling method and system, which enables differentiated bandwidth control of protocol traffic with different transmission characteristics in AI storage scenarios where remote direct memory access and transmission control protocols coexist, so as to give full play to their respective transmission performance. Furthermore, it enables fast and transparent IO path switching to upper-layer applications when a single transmission plane fails, thereby avoiding business interruption and data inconsistency.

[0004] This application discloses an RDMA / TCP dual-transmission plane cooperative QoS scheduling method, including: Four token buckets are established for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. A first time slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time slice granularity and a second smoothing suppression strength are configured for the transmission control protocol plane. The first time slice granularity is larger than the second time slice granularity, and the first smoothing suppression strength is smaller than the second smoothing suppression strength. Collect the flow control trigger count, explicit congestion notification flag count, and retry count of the remote direct memory access plane, as well as the round-trip time, retransmission count, and congestion window size of the transmission control protocol plane, and adjust the corresponding token bucket filling rate and suppression parameters based on the collected data; Admission control for IO requests is performed based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion.

[0005] Optionally, four token buckets can be created for the same tenant, including: In the tenant data structure, separate token bucket structures are allocated for remote direct memory access read, remote direct memory access write, transmission control protocol read, and transmission control protocol write. Each token bucket structure contains a capacity limit field, a current available quota field, and a fill rate field, and concurrent access is protected by an independent spin lock.

[0006] Optionally, a first time-slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time-slice granularity and a second smoothing suppression strength are configured for the transport control protocol plane, including: Set the first time slice granularity of the remote direct memory access plane to 5 milliseconds and the first smoothing suppression strength to 50. Set the second time slice granularity of the transmission control protocol plane to 2 milliseconds and the second smoothing suppression strength to 80.

[0007] Optionally, the number of flow control triggers, explicit congestion notification markers, and retries of the remote direct memory access plane, as well as the round-trip time, retransmission count, and congestion window size of the transmission control protocol plane, are collected, including: The number of flow control triggers, the number of explicit congestion notifications, and the number of retries in the remote direct memory access plane are read using the performance counters in the network card driver. The round-trip time, retransmission count, and congestion window size of the Transmission Control Protocol (TCP) plane are read from the TCP stack. The round-trip time includes the average round-trip time and the round-trip time variance.

[0008] Optionally, the filling rate and suppression parameters of the corresponding token bucket can be adjusted based on the collected data, including: When the number of flow control triggers of the remote direct memory access plane increases beyond a preset threshold or the number of retries exceeds a preset retry threshold within 1 second, the filling rate of the remote direct memory access token bucket is reduced and its smoothing suppression strength is increased. The remote direct memory access token bucket includes a remote direct memory access read token bucket and a remote direct memory access write token bucket. When the average round-trip time of the Transmission Control Protocol (TCP) plane exceeds a preset multiple of the historical baseline value and the number of retransmissions continues to increase, the filling rate of the TCP token bucket is reduced. The TCP token bucket includes a TCP read token bucket and a TCP write token bucket.

[0009] Optionally, reduce the fill rate of the remote direct memory access token bucket, including adjusting the fill rates of the remote direct memory access read token bucket and the remote direct memory access write token bucket to 70% of their current values; The improvement of its smoothing suppression strength includes: increasing the smoothing suppression strength to 80.

[0010] Optionally, a failure is detected in the remote direct memory access plane or the transport control protocol plane, including: When the number of retries in the remote direct memory access plane exceeds 100 within 5 seconds, or when the number of flow control triggers continues to rise and the effective throughput drops to less than 30% of the baseline throughput, the remote direct memory access plane is considered to be faulty. A transmission control protocol plane (TCP) failure is determined when the number of retransmissions exceeds 50 within 5 seconds, or when the congestion window size remains at 1 and the round-trip delay is greater than 100 milliseconds.

[0011] Optionally, newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling, while ongoing IO requests that have been admitted through the fault plane token bucket and issued before the failure occur continue to execute on the fault plane path until completion, including: Set an atomic switching flag in the tenant structure and update the current priority transport plane identifier to the healthy plane; Before a new IO request is admitted to the token bucket, the atomic switching flag is read, the corresponding token bucket is found according to the health plane indicated by the atomic switching flag, the quota is applied for, and the protocol type of the plane is marked. For ongoing I / O requests that have been admitted through the fault plane token bucket and have been sent to the network protocol stack at the switching time, their original transmission path is retained. When the ongoing I / O request is completed, the consumed quota is deducted from the fault plane token bucket through a statistical hook. The migration ends when all in-process I / O requests on the fault plane are completed.

[0012] Optionally, before binding newly arriving IO requests to the health plane, the following steps may also be included: Check if the logical block address range of the newly arrived IO request overlaps with the logical block address range of the IO request in progress on the fault plane. If there is an overlap, postpone the distribution of the newly arrived IO request until the overlapping IO request in progress is completed before distributing it to the healthy plane.

[0013] This application also discloses an RDMA / TCP dual-transmission plane cooperative QoS scheduling system, including: A dual-plane token bucket isolation module is used to establish four token buckets for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. A first time slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time slice granularity and a second smoothing suppression strength are configured for the transmission control protocol plane. The first time slice granularity is greater than the second time slice granularity, and the first smoothing suppression strength is less than the second smoothing suppression strength. The transport layer feedback closed-loop module is used to collect the number of flow control triggers, the number of explicit congestion notification markers, and the number of retries of the remote direct memory access plane, as well as the round-trip delay, the number of retransmissions, and the congestion window size of the transport control protocol plane, and adjust the filling rate and suppression parameters of the corresponding token bucket according to the collected data. The cross-plane migration module is used to perform admission control on IO requests based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, the newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion.

[0014] As can be seen from the above technical solution, this application establishes independent token buckets for remote direct memory access (RDM) reads and writes and transmission control protocol (TCP) reads and writes for the same tenant at the block device layer. It configures a larger time slice granularity and a lower smoothing suppression strength for the RDM plane, and a smaller time slice granularity and a higher smoothing suppression strength for the TCP plane. This ensures that high-throughput, low-latency RDM requests are no longer interfered with by small-granular TCP requests when requesting quotas. The token bucket filling cycle matches the high-throughput characteristics of kernel bypass. Meanwhile, the TCP plane's token buckets are issued at a finer-grained rate, coupled with its congestion-adaptive window adjustment behavior, preventing bandwidth encroachment between protocols from the source. During IO scheduling, the number of flow control triggers, explicit congestion notification markings, and retries for RDM, as well as the round-trip latency, retransmission count, and congestion window size for TCP, are continuously collected. When the transport layer indicators of any plane deteriorate, the filling rate of the corresponding token bucket is synchronously reduced. The rate control intensity is increased to ensure that the rate control direction of the block layer is aligned with the adaptive rate reduction direction of the transport layer. This eliminates the dual suppression caused by the information separation between the network layer and the block layer, ensuring effective throughput in congested scenarios. Furthermore, when a hardware failure or unrecoverable congestion is determined to occur in a certain plane based on the aforementioned transport layer indicators, an atomic switching flag is set at the block layer to directly bind subsequent newly arriving IO requests to the token bucket of the healthy plane. Ongoing IO requests already sent to the faulty plane maintain their original transmission paths until they are completed naturally. This request-granular path binding and natural exhaustion strategy does not require canceling or retransmitting any ongoing IO requests. Upper-layer applications only perceive microsecond-level latency jitter in individual IO requests and do not receive error returns. The entire migration process is completed in sub-seconds. In addition, the token bucket of the healthy plane is pre-expanded before migration, and the logical block address overlap between newly arriving IO requests and ongoing IO requests is detected, ensuring data consistency and smooth bandwidth transition during the switching process. Through the synergy of the above components, protocol-differentiated rate control, network state adaptive adjustment, and sub-second transparent fault migration are achieved in a hybrid transmission environment of remote direct memory access and transmission control protocol, thereby improving the overall throughput stability and business continuity of AI training storage. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1A flowchart of an embodiment of the RDMA / TCP dual-transmission plane cooperative QoS scheduling method provided in this application; Figure 2 A flowchart for configuring time slice granularity and smoothing suppression strength for the remote direct memory access plane and the transfer control protocol plane provided in this application; Figure 3 for Figure 1 Sub-flowcharts of S200; Figure 4 The flowchart provided in this application shows how to bind newly arriving IO requests to the token bucket of the health plane for scheduling, while maintaining the ongoing IO requests that have been admitted and issued by the token bucket of the fault plane before the failure occurs to continue to be executed on the fault plane path until completion. Figure 5 The structural block diagram of the RDMA / TCP dual-transmission plane collaborative QoS scheduling system provided in this application is shown. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] Please refer to Figure 1 , Figure 1 A flowchart of an embodiment of the RDMA / TCP dual-transmission plane cooperative QoS scheduling method provided in this application is shown below. Figure 1 As shown, it includes the following steps: S100, four token buckets are established for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. A first time slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time slice granularity and a second smoothing suppression strength are configured for the transmission control protocol plane. The first time slice granularity is greater than the second time slice granularity, and the first smoothing suppression strength is less than the second smoothing suppression strength.

[0019] The operating system kernel block device layer, located between the file system layer and the storage device driver layer, is the necessary processing path for all block device I / O requests. Deploying scheduling logic at this layer enables unified management of I / O requests from all upper-layer applications without requiring modifications to application code or file system implementations, offering excellent compatibility and transparency.

[0020] In this context, "same tenant" refers to users or application instances that share the same set of storage resource quotas. In a multi-tenant environment, each tenant has an independent resource isolation domain, and the scheduling parameters and token bucket status of different tenants do not interfere with each other, which can effectively avoid resource contention between tenants.

[0021] In this embodiment, the Remote Direct Memory Access (RDA) read token bucket is used to manage the bandwidth usage of all read I / O requests sent through the RDA protocol, the RDA write token bucket is used to manage the bandwidth usage of all write I / O requests sent through the RDA protocol, the Transmission Control Protocol (TCP) read token bucket is used to manage the bandwidth usage of all read I / O requests sent through the TCP protocol, and the TCP write token bucket is used to manage the bandwidth usage of all write I / O requests sent through the TCP protocol.

[0022] In existing technologies, a single token bucket is typically maintained for each tenant at the block device layer, with all types of I / O requests consuming the same token quota. This approach fails to consider the differences in transmission characteristics between Remote Direct Memory Access (RDM) and Transmission Control Protocol (TCP), causing large-granularity I / O requests from RDM to be frequently interrupted by small-granularity I / O requests from TCP. For example, when a 1MB RDM write I / O request requests a token, if the currently available token quota in the token bucket is less than 1MB, the I / O request will be split into multiple smaller I / Os for transmission. Each smaller I / O needs to wait for the token bucket to fill with enough tokens before it can be sent. This segmentation increases I / O processing overhead and transmission latency, while also reducing the effective throughput of RDM.

[0023] Compared to existing technologies, one improvement of this application lies in establishing four independent token buckets to completely isolate traffic from different protocols and directions. This ensures that large-granularity I / O requests for Remote Direct Memory Access (RDA) are not interfered with by small-granularity I / O requests from the Transmission Control Protocol (TCP) when requesting quotas, allowing them to acquire enough tokens for complete transmission at once and fully leverage the high throughput of RDA. The reason for managing read and write requests separately is that they differ significantly in transmission characteristics and impact on system performance. Specifically, read requests are typically more latency-sensitive because upper-layer applications often need to wait for read data to return before continuing subsequent calculations; increased read latency directly leads to a decrease in application performance. Write requests, on the other hand, can be merged and batch-processed through caching mechanisms in many scenarios, exhibiting relatively lower latency sensitivity but requiring higher throughput. Completely isolating the four traffic types prevents one type of traffic from excessively consuming bandwidth and affecting the service quality of other types, achieving more granular bandwidth management.

[0024] In this context, time slice granularity refers to the minimum time interval for the token bucket to perform token filling operations. A larger time slice granularity means the token bucket will fill more tokens at once over a longer time interval, suitable for handling large-granularity, high-throughput traffic, reducing the number of token filling operations and lowering system overhead. A smaller time slice granularity means the token bucket will fill fewer tokens over a shorter time interval, suitable for handling small-granularity, frequent bursts of traffic, smoothing the traffic transmission rate and avoiding traffic bursts. Smoothing suppression strength refers to the extent to which the token bucket tightens quota issuance when it detects burst traffic. A lower smoothing suppression strength means the token bucket has a higher tolerance for burst traffic, allowing more tokens to be issued in a short period, improving bandwidth utilization. A higher smoothing suppression strength means the token bucket has a lower tolerance for burst traffic, quickly tightening quota issuance to avoid traffic overload and ensuring system stability. This application does not limit the value of these values; those skilled in the art can adjust them according to the actual network environment and business needs.

[0025] Remote Direct Memory Access (RDMA) protocols employ kernel bypass technology, meaning data transfer does not require the operating system kernel's involvement. Data can be copied directly between user-space memory and the network interface card (NIC), resulting in extremely low latency and very high throughput. RDMA traffic typically consists of large-granularity sequential I / O, such as dataset loading and checkpoint saving during AI training, where a single I / O request can reach hundreds of KB or even several MB. To fully leverage the high throughput of RDMA, a larger time-slice granularity is needed. This allows the token bucket to fill with enough tokens at once to meet the quota requirements of large-granularity I / O, preventing it from being frequently split into multiple smaller I / Os. If the time-slice granularity is too small, the token bucket can only fill with a limited number of tokens at a time, forcing large-granularity I / Os to wait for multiple token fillings to obtain sufficient quota. This increases I / O processing overhead and transmission latency, while also reducing the effective throughput of RDMA. Meanwhile, since the Remote Direct Memory Access Protocol itself has a link-layer flow control mechanism that can actively reduce the transmission rate when network congestion occurs, it can be configured with a lower smoothing suppression strength to allow a certain degree of traffic bursts in order to improve bandwidth utilization.

[0026] Transmission Control Protocol (TCP) is a kernel-based reliable transport protocol. Data transmission requires processing by the operating system kernel's protocol stack, including packet segmentation, acknowledgment retransmission, and congestion control. TCP traffic typically consists of small-granularity random I / O, such as management commands and metadata operations, with individual I / O requests usually ranging from a few KB to tens of KB. TCP's congestion control mechanism is highly sensitive to traffic bursts; a large surge in traffic within a short period can cause the TCP congestion window to shrink drastically, leading to a significant drop in throughput. To avoid this, a smaller time-slice granularity needs to be configured for the TCP plane, allowing the token bucket to issue tokens at a finer granularity and smooth the transmission rate. If the time-slice granularity is too large, the token bucket will fill with a large number of tokens at once, resulting in a large number of TCP packets being sent to the network in a short period. This triggers the TCP congestion control mechanism, causing the congestion window to shrink and throughput to decrease. At the same time, it is configured with a high smoothing suppression strength, which can quickly tighten the quota allocation when a traffic burst is detected, prevent the burst traffic from triggering the congestion control mechanism of the transmission control protocol, and ensure the stability of the transmission control protocol connection.

[0027] For example, the first time slice granularity can range from 3 milliseconds to 10 milliseconds, and the first smoothing suppression intensity can range from 30 to 70. The second time slice granularity can range from 1 millisecond to 4 milliseconds, and the second smoothing suppression intensity can range from 60 to 90.

[0028] S200: Collect the number of flow control triggers, the number of explicit congestion notification markers, and the number of retries of the remote direct memory access plane, as well as the round-trip time, the number of retransmissions, and the congestion window size of the transmission control protocol plane, and adjust the filling rate and suppression parameters of the corresponding token bucket based on the collected data.

[0029] S300 performs admission control on IO requests based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion.

[0030] Compared with related technologies, this application establishes independent token buckets for remote direct memory access (RDM) reads and writes and transmission control protocol (TCP) reads and writes for the same tenant at the block device layer. It configures a larger time slice granularity and a lower smoothing suppression strength for the RDM plane, and a smaller time slice granularity and a higher smoothing suppression strength for the TCP plane. This ensures that high-throughput, low-latency RDM requests are no longer interfered with by small-granular TCP requests when requesting quotas. The token bucket filling cycle matches the high-throughput characteristics of kernel bypass. The TCP plane's token buckets are issued at a finer-grained rate, coupled with its congestion-adaptive window adjustment behavior, preventing bandwidth encroachment between protocols from the source. During IO scheduling, the number of RDM flow control triggers, explicit congestion notification markings, and retries are continuously collected, along with the TCP round-trip latency, retransmission count, and congestion window size. When the transport layer indicators of any plane deteriorate, the filling rate of the corresponding token bucket is synchronously reduced and the suppression strength is increased, thus enabling rate control at the block layer. The direction of the adaptive rate reduction is aligned with that of the transport layer, thereby eliminating the dual suppression caused by the information separation between the network layer and the block layer, ensuring effective throughput in congested scenarios. In addition, when a hardware failure or unrecoverable congestion is determined to occur in a certain plane based on the above transport layer indicators, an atomic switching flag is set in the block layer to directly bind subsequent new IO requests (which can also be represented as new IO requests) to the token bucket of the healthy plane. Meanwhile, ongoing IO requests that have been sent to the faulty plane maintain their original transmission paths until they are completed naturally. This request-granular path binding and natural exhaustion strategy does not require canceling or retransmitting any ongoing IO requests. Upper-layer applications only perceive microsecond-level latency jitter in individual IO requests and do not receive error returns. The entire migration process is completed in sub-seconds. Through the synergy of the above parts, protocol-differentiated rate control, network state adaptive adjustment, and sub-second transparent fault migration are achieved in a hybrid transmission environment of remote direct memory access and transmission control protocol, which improves the overall throughput stability and business continuity of AI training storage.

[0031] Optionally, four token buckets are created for the same tenant, including: allocating independent token bucket structures in the tenant data structure for remote direct memory access read, remote direct memory access write, transmission control protocol read, and transmission control protocol write respectively. Each token bucket structure contains a capacity limit field, a current available quota field, and a fill rate field, and is protected against concurrent access by an independent spin lock.

[0032] The tenant data structure is a data structure in the operating system kernel used to store tenant-related information. Each tenant corresponds to a unique instance of the tenant data structure. The tenant data structure typically includes the tenant's identifier, resource quotas, permission information, and statistical information. Allocating independent token bucket structures within the tenant data structure allows for centralized management of the token bucket's state and other tenant information, facilitating resource scheduling and isolation by the system. When a tenant is created, the system automatically creates four independent token bucket structures for that tenant and initializes the corresponding parameters; when a tenant is destroyed, the system automatically releases the memory resources occupied by these four token bucket structures.

[0033] Specifically, the maximum capacity field in the token bucket structure specifies the maximum number of tokens that the token bucket can hold. When the available quota in the token bucket reaches the maximum capacity, subsequent tokens will be discarded. The value of the maximum capacity determines the maximum burst traffic the token bucket can tolerate; a larger maximum capacity allows for a larger burst of traffic, but may also increase the risk of network congestion. The current available quota field records the number of tokens currently available in the token bucket. When an IO request requests a token, the corresponding quota is deducted from this field; when the token bucket is filled, the corresponding quota is added to this field. The fill rate field specifies the number of tokens filled per unit of time. This field determines the average bandwidth that the token bucket can provide; a higher fill rate provides a larger average bandwidth.

[0034] For example, the maximum capacity of the token bucket can range from 500 token units to 2000 token units, and the fill rate can range from 100 token units per millisecond to 1000 token units per millisecond. Each token unit can correspond to a fixed-size data block, such as 4KB. Thus, a fill rate of 100 token units per millisecond is equivalent to 400MB of bandwidth per second. This application does not limit these values, and those skilled in the art can adjust them according to actual network bandwidth and service requirements. For example, for a 100GbE network, the fill rate can be set to 3000 token units per millisecond, corresponding to 12GB of bandwidth per second; for a 25GbE network, the fill rate can be set to 750 token units per millisecond, corresponding to 3GB of bandwidth per second.

[0035] Furthermore, each token bucket structure is protected against concurrent access using an independent spinlock. A spinlock is a lightweight synchronization mechanism where, when a CPU core attempts to acquire an already held spinlock, it loops and waits until the lock is released. Spinlocks have very low overhead and are suitable for protecting operations that can be completed in a short time. Assigning an independent spinlock to each token bucket avoids lock contention between different token buckets. For example, while one CPU core is operating on a remote direct memory access read token bucket, another CPU core can simultaneously operate on a transfer control protocol write token bucket without lock contention. This design significantly improves the system's concurrent processing capabilities, especially in multi-CPU core systems. If a single spinlock were used to protect all four token buckets, severe lock contention would occur when multiple CPU cores operate on different token buckets simultaneously, leading to a decrease in the system's concurrent processing capabilities.

[0036] Furthermore, the token bucket filling operation can be performed by a separate kernel thread. This kernel thread is periodically woken up according to a preset time slice granularity, then iterates through all token buckets of all tenants, filling each token bucket with the appropriate number of tokens. The number of tokens filled is equal to the filling rate multiplied by the time slice granularity. For example, if the filling rate is 200 token units per millisecond and the time slice granularity is 5 milliseconds, then the number of tokens filled each time is 1000 token units. If the available quota after filling exceeds the capacity limit, the excess will be discarded. The kernel thread priority can be set to a high level to ensure that the token filling operation is executed in a timely manner, avoiding token shortages.

[0037] Those skilled in the art will understand that existing single token buckets typically use only one spinlock for protection. When multiple CPU cores simultaneously request tokens, severe lock contention occurs, leading to a decrease in the system's concurrent processing capability. This application, however, by assigning an independent spinlock to each token bucket, refines the lock granularity from the tenant level to the traffic type level, effectively reducing lock contention and improving the system's concurrent processing capability. Simultaneously, independent token buckets allow the system to configure different capacity limits and fill rates for different types of traffic, achieving more granular bandwidth management.

[0038] In some embodiments, please refer to Figure 2 , Figure 2 The flowchart provided in this application for configuring time slice granularity and smoothing suppression strength for the remote direct memory access plane and the transport control protocol plane is as follows: Figure 2 As shown, the remote direct memory access plane is configured with a first time slice granularity and a first smoothing suppression strength, and the transport control protocol plane is configured with a second time slice granularity and a second smoothing suppression strength, including: S110, set the first time slice granularity of the remote direct memory access plane to 5 milliseconds, and the first smoothing suppression strength to 50; S120, set the second time slice granularity of the transmission control protocol plane to 2 milliseconds, and the second smoothing suppression strength to 80.

[0039] It should be noted that the above values ​​are the optimal combination verified through extensive experiments, achieving good performance in most AI storage scenarios. A 5-millisecond time slice granularity for the Remote Direct Memory Access (RDA) plane ensures the token bucket is filled with enough tokens at once to meet the needs of large-granularity I / O, without causing token filling delays due to excessively long time slices. If the time slice granularity is set to 10 milliseconds, the token bucket will only be filled with tokens every 10 milliseconds. When large-granularity I / O requests tokens, if the available quota is insufficient, it will require waiting up to 10 milliseconds to obtain enough tokens, increasing I / O latency. If the time slice granularity is set to 3 milliseconds, the token bucket will be filled with tokens every 3 milliseconds. While this reduces I / O wait time, it increases the number of token filling operations, increasing system CPU overhead. A 5-millisecond time slice granularity achieves a good balance between I / O latency and system overhead. A smoothing suppression strength of 50 means that when the traffic to the RDA plane exceeds the average rate, the token bucket will tighten quota issuance at a moderate rate, avoiding excessive traffic bursts without overly restricting normal traffic fluctuations. If the smoothing suppression strength is set to 30, the token bucket has a high tolerance for burst traffic, which may lead to excessive burst traffic entering the network and causing network congestion. If the smoothing suppression strength is set to 70, the token bucket has a low tolerance for burst traffic, which may limit normal traffic fluctuations and reduce bandwidth utilization.

[0040] For example, a 2-millisecond time slice granularity allows for sufficiently fine-grained token issuance for the Transmission Control Protocol (TCP) plane, smoothing the transmission rate and preventing large bursts of traffic in a short period from triggering TCP congestion control mechanisms. If the time slice granularity is set to 4 milliseconds, the token bucket will only be filled with tokens every 4 milliseconds, potentially leading to a large number of TCP messages being sent to the network in a short time, triggering TCP congestion control mechanisms. If the time slice granularity is set to 1 millisecond, the token bucket will be filled with tokens every 1 millisecond, which, while better smoothing traffic, increases the number of token filling operations and increases system CPU overhead. A 2-millisecond time slice granularity strikes a good balance between traffic smoothing and system overhead. A smoothing suppression strength of 80 means that when the traffic on the TCP plane exceeds the average rate, the token bucket will quickly tighten its quota issuance, effectively suppressing traffic bursts and ensuring the stability of TCP connections. If the smoothing suppression strength is set to 60, the token bucket has a high tolerance for burst traffic, which may not effectively suppress bursts, leading to contraction of the TCP congestion window. If the smoothing suppression strength is set to 90, the token bucket has too low tolerance for burst traffic, which may limit normal traffic fluctuations and reduce bandwidth utilization.

[0041] Specifically, the smoothing suppression strength works as follows: the system records the token consumption rate of each token bucket over a period of time. When the current token consumption rate exceeds a certain percentage of the filling rate, the system adjusts the token issuance rate according to the smoothing suppression strength. The higher the smoothing suppression strength, the larger the adjustment. For example, when the smoothing suppression strength is 50, if the current token consumption rate exceeds 120% of the filling rate, the actual token issuance rate of the token bucket will be adjusted to 90% of the filling rate. When the smoothing suppression strength is 80, if the current token consumption rate exceeds 120% of the filling rate, the actual token issuance rate of the token bucket will be adjusted to 70% of the filling rate. This application does not limit the value of this value; those skilled in the art can adjust it according to the actual network environment and business needs.

[0042] Furthermore, time slice granularity and smoothing suppression strength are dynamically adjustable parameters. The system can automatically adjust the values ​​of these two parameters based on the current network load and service type. For example, when the system detects that traffic in the Remote Direct Memory Access plane (RDBMP) is mainly composed of small-granularity random I / O requests, it can appropriately reduce the time slice granularity of the RDBMP and increase the smoothing suppression strength to better adapt to this traffic characteristic. When the system detects that traffic in the Transmission Control Protocol (TCP) plane is mainly composed of large-granularity sequential I / O requests, it can appropriately increase the time slice granularity of the TCP plane and decrease the smoothing suppression strength to improve bandwidth utilization. The system can also adjust the values ​​of these two parameters according to different times of the day; for example, it can increase the smoothing suppression strength during peak traffic periods to ensure system stability and decrease it during off-peak traffic periods to improve bandwidth utilization.

[0043] Compared to existing technologies, which typically use fixed time slice granularity and smoothing suppression strength to configure the token bucket without considering the differences in transmission characteristics of different protocols, this approach cannot simultaneously meet the needs of remote direct memory access and transmission control protocols, often resulting in the performance of one of the protocols being affected. In contrast, this application configures differentiated time slice granularity and smoothing suppression strength for different protocol planes, enabling the token bucket behavior to match the transmission characteristics of the protocols, thereby fully leveraging the advantages of both protocols.

[0044] Optional, please refer to Figure 3 , Figure 3 for Figure 1 Sub-flowcharts of S200, such as Figure 3 As shown, the data collection includes the number of flow control triggers, explicit congestion notification flags, and retries of the remote direct memory access plane, as well as the round-trip time, retransmission count, and congestion window size of the transmission control protocol plane, including: S210 reads the number of flow control triggers, the number of explicit congestion notifications, and the number of retries for the remote direct memory access plane through the performance counters in the network card driver.

[0045] The performance counters in the network interface card (NIC) driver are a set of registers provided by the hardware NIC to record various statistical information during NIC operation. These counters are automatically updated by the hardware without software intervention, so reading these counters does not introduce additional data copying overhead and has minimal impact on system performance. Remote direct memory access (RDA) NICs typically provide a rich set of performance counters, including the number of flow control triggers, the number of explicit congestion notification markers, the number of retries, the number of bytes sent, the number of bytes received, and the number of errors. By reading these counters, the system can understand the real-time operating status of the RDA and promptly detect network congestion and hardware failures.

[0046] The flow control trigger count refers to the number of times a remote direct memory access network card (RDBIT) triggers flow control due to a full receiver buffer. When the receiver's buffer cannot hold more data, it sends a flow control frame to the sender, notifying it to pause data transmission. An increase in the number of flow control triggers usually indicates network congestion or insufficient processing capacity at the receiver.

[0047] The explicit congestion notification flag count refers to the number of packets with explicit congestion notification flags received by the remote direct memory access network interface card (DDRNIC). Explicit congestion notification is a network congestion indication mechanism. When a network device detects congestion, it sets an explicit congestion notification flag in forwarded packets, notifying the sender to reduce its transmission rate. An increase in the explicit congestion notification flag count is a direct signal that network congestion has occurred.

[0048] The retry count refers to the number of times a remote direct memory access network card retransmits data because it has not received an acknowledgment message from the receiving end. An increase in the retry count usually indicates packet loss or a deterioration in network link quality.

[0049] S220, read the round-trip time, retransmission count and congestion window size of the Transmission Control Protocol (TCP) plane through the TCP protocol stack, wherein the round-trip time includes the average round-trip time and the round-trip time variance.

[0050] Specifically, the Transmission Control Protocol (TCP) stack maintains detailed statistics for each connection, including round-trip time (RTT), retransmission count, congestion window size, and slow start threshold. These statistics are stored in TCP control blocks, with one TCP control block instance corresponding to each TCP connection. The system can collect statistics for each connection by traversing all TCP control blocks, and then calculate the average RTT, RTT variance, total retransmission count, and average congestion window size for the entire TCP plane.

[0051] Round-trip time (RTT) refers to the time elapsed from sending a message to receiving its acknowledgment. An increase in RTT usually indicates network congestion or increased link latency. RTT variance refers to the degree of fluctuation in RTT; a larger variance indicates more unstable network latency.

[0052] The retransmission count refers to the number of times the Transmission Control Protocol (TCP) retransmits data because it has not received an acknowledgment message. An increase in the retransmission count usually indicates that packet loss has occurred in the network.

[0053] The congestion window size refers to the maximum amount of data that the Transmission Control Protocol (TCP) can send without receiving an acknowledgment message. The congestion window size is a core parameter of the TCP congestion control mechanism; it shrinks drastically when network congestion occurs.

[0054] In this embodiment, the system periodically collects the aforementioned statistical indicators, with the collection period ranging from 100 milliseconds to 1 second. The selection of the collection period requires a trade-off between real-time performance and system overhead. A shorter collection period provides more real-time network status information but increases system CPU overhead. A longer collection period reduces system CPU overhead but leads to a lag in network status information. This application does not limit this value; those skilled in the art can adjust it according to actual system performance and requirements. For example, for systems with high real-time requirements, the collection period can be set to 100 milliseconds; for systems with high system overhead requirements, the collection period can be set to 1 second.

[0055] Those skilled in the art will understand that existing block device layer quality of service mechanisms typically do not collect statistical information from the transport layer, but instead use fixed token bucket parameters for bandwidth control. This approach results in a lack of information pathways between the block layer's rate control and the actual state of the transport layer. When network congestion occurs, the transport layer has already proactively reduced the transmission rate through congestion control or flow control mechanisms, but the block device layer's token bucket continues to fill with tokens at a fixed rate, continuing to send data to the network. This further exacerbates network congestion and also causes token accumulation at the block layer, preventing subsequent IO requests from obtaining tokens in a timely manner. This creates a double inhibition, leading to a significant decrease in effective throughput.

[0056] This application continuously collects real-time statistical indicators from the transport layer, enabling timely understanding of the network's operational status. Based on these indicators, it dynamically adjusts the token bucket parameters, ensuring that the block layer's rate control and the transport layer's adaptive rate reduction move in the same direction. This eliminates the dual suppression and improves the effective throughput in congested scenarios.

[0057] Optional, continue to refer to Figure 3 Adjust the filling rate and suppression parameters of the corresponding token bucket based on the collected data, including: S230, when the number of flow control triggers of the remote direct memory access plane increases beyond a preset threshold or the number of retries exceeds a preset retry threshold within 1 second, the filling rate of the remote direct memory access token bucket is reduced and its smoothing suppression strength is increased. The remote direct memory access token bucket includes a remote direct memory access read token bucket and a remote direct memory access write token bucket.

[0058] The preset number of attempts threshold can range from 5 to 20, and the preset retry threshold can range from 10 to 50. The historical baseline value refers to the average round-trip time (RTT) value collected by the system under normal network conditions. The system collects RTT data for a period of time under idle or lightly loaded network conditions and calculates the average value as the historical baseline value. The preset multiple can range from 1.5 to 3 times. This application does not limit this value; those skilled in the art can adjust it according to the actual network environment and business needs.

[0059] For example, when the number of flow control triggers on the Remote Direct Memory Access (RDA) plane increases by more than 10 times per second, it indicates severe network congestion. The receiver's buffer is frequently filled, making it unable to process data from the sender in a timely manner. In this case, the system needs to reduce the filling rate of the RDA token bucket and decrease the amount of data sent to the network to alleviate congestion. Simultaneously, increasing the smoothing suppression strength can further suppress traffic bursts and prevent the congestion from worsening. When the number of retries on the RDA plane exceeds 30 times per second, it indicates severe packet loss or link quality degradation. The same measures should be taken to reduce the sending rate.

[0060] S240, when the average round-trip time of the transmission control protocol plane exceeds a preset multiple of the historical baseline value and the number of retransmissions continues to increase, the filling rate of the transmission control protocol token bucket is reduced. The transmission control protocol token bucket includes a transmission control protocol read token bucket and a transmission control protocol write token bucket.

[0061] Specifically, when the average round-trip time (RTT) of the Transmission Control Protocol (TCP) plane exceeds twice the historical baseline, and the number of retransmissions shows an increasing trend over two consecutive sampling periods, it indicates that the TCP connection has entered a congested state. The TCP congestion control mechanism automatically shrinks the congestion window and reduces the transmission rate. At this time, the system needs to synchronously reduce the filling rate of the TCP token bucket to match the actual transmission rate of the block layer. If the block layer continues to send data to the transport layer at the original rate, it will cause the transport layer's transmission queue to accumulate, increasing I / O latency, and may even cause the transport layer's buffer to overflow, resulting in data loss. It is important to note that the requirement for a continuous increase in the number of retransmissions is to avoid misjudging temporary network fluctuations as congestion. If only a single sampling shows an increase in the number of retransmissions, but the average RTT does not change significantly, it may only be a temporary network fluctuation, and there is no need to adjust the token bucket parameters.

[0062] Furthermore, once network congestion subsides, the system gradually restores the token bucket filling rate and smoothing suppression strength. The recovery step size can be set to increase the current filling rate by 5% to 15% per acquisition cycle. The recovery process stops once the token bucket filling rate returns to the baseline value. This gradual recovery method avoids sudden increases in the sending rate that could trigger network congestion again. For example, if the filling rate is adjusted to 70% of the baseline value, and the recovery step size is 10%, the system will increase the filling rate by 7% of the baseline value in each acquisition cycle until the filling rate returns to the baseline value.

[0063] Compared to existing technologies, where token bucket parameters are typically statically configured and not dynamically adjusted based on real-time network conditions, this approach cannot adapt to changes in network status. It leads to double suppression during network congestion and fails to fully utilize available bandwidth during network downtime. This application, however, establishes a transport layer feedback loop to dynamically adjust token bucket parameters based on real-time collected transport layer statistics, enabling block-layer rate control to match the actual network state. When network congestion occurs, the token bucket filling rate is synchronously reduced to avoid double suppression; when the network is idle, the token bucket filling rate is gradually restored to fully utilize available bandwidth. This adaptive adjustment mechanism significantly improves network bandwidth utilization and system throughput stability.

[0064] Optionally, reducing the filling rate of the remote direct memory access token bucket includes adjusting the filling rate of the remote direct memory access read token bucket and the remote direct memory access write token bucket to 70% of the current value; increasing its smoothing suppression strength includes increasing the smoothing suppression strength to 80.

[0065] Regarding the fill rate, if the adjustment ratio is too low, such as to 50% of the current value, it will cause an excessive drop in the transmission rate, affecting the normal operation of services. If the adjustment ratio is too high, such as to 90% of the current value, it will not effectively alleviate network congestion and may even further worsen the congestion situation. Meanwhile, increasing the smoothing suppression strength to 80 can effectively suppress traffic bursts and prevent further exacerbation of congestion. A smoothing suppression strength of 80 is the same as the default smoothing suppression strength in the transmission control protocol plane, providing sufficient traffic suppression capability during network congestion.

[0066] For example, when the baseline fill rate of the remote direct memory access write token bucket is 500 tokens per millisecond, if network congestion is detected, the system will adjust its fill rate to 350 tokens per millisecond. Simultaneously, the smoothing suppression strength will be increased from 50 to 80. This significantly reduces the token bucket sending rate, effectively alleviating network congestion. Once network congestion is relieved, the system will gradually restore the fill rate in steps of 35 tokens per millisecond per sampling cycle until it returns to the baseline value of 500 tokens per millisecond. At the same time, the smoothing suppression strength will also gradually return to the baseline value of 50.

[0067] In some embodiments, the same adjustment ratio can be used for the remote direct memory access read token bucket and the remote direct memory access write token bucket, or different adjustment ratios can be used. For example, since write requests are generally less sensitive to latency, a larger adjustment ratio can be used for the write token bucket and a smaller adjustment ratio for the read token bucket to prioritize the quality of service for read requests. For example, the fill rate of the write token bucket can be adjusted to 60% of its current value, while the fill rate of the read token bucket can be adjusted to 80% of its current value. This application does not limit these values, and those skilled in the art can adjust them according to actual business needs.

[0068] For example, the system can adjust the filling rate according to the severity of congestion. For instance, when the number of flow control triggers increases by more than 20 times per second, the filling rate can be adjusted to 50% of the current value; when the number of flow control triggers increases by more than 10 but less than 20 times per second, the filling rate can be adjusted to 70% of the current value; and when the number of flow control triggers increases by more than 5 but less than 10 times per second, the filling rate can be adjusted to 90% of the current value. This tiered adjustment method can more finely adapt to different levels of network congestion, alleviating congestion while maintaining a high throughput as much as possible.

[0069] Compared to existing technologies, even those with mechanisms for dynamically adjusting token bucket parameters typically only adjust them based on the block layer's I / O queue length or waiting time, without considering the actual state of the transport layer. However, an increase in the block layer's I / O queue length could be due to network congestion or insufficient processing power of the storage device. Relying solely on block layer information cannot accurately determine the network state, resulting in unsatisfactory adjustment effects. In contrast, this application directly adjusts the token bucket parameters based on transport layer statistical indicators, accurately reflecting the actual network state, thus achieving more precise and effective adjustments.

[0070] Optionally, a failure is detected in the remote direct memory access plane or the transport control protocol plane, including: A1. When the number of retries of the remote direct memory access plane exceeds 100 within 5 seconds, or the number of flow control triggers continues to rise and the effective throughput drops to less than 30% of the baseline throughput, the remote direct memory access plane is determined to be faulty. The above fault diagnosis criteria have been verified through extensive experiments and can accurately distinguish between temporary network fluctuations and permanent hardware failures or unrecoverable congestion. The 5-second statistical window filters out short-term network jitter, avoiding false positives. The 100-retry threshold means that there is an average of one retransmission every 25 milliseconds within 5 seconds, which far exceeds the retransmission rate of a normal network, indicating a serious network problem. Effective throughput dropping below 30% of baseline throughput indicates that the remote direct memory access plane can no longer provide normal service quality; continued use of this plane will lead to a severe degradation in business performance.

[0071] For example, baseline throughput refers to the maximum effective throughput that the remote direct memory access plane (RDP) can achieve under normal network conditions. The system performs bandwidth tests under idle or lightly loaded network conditions to measure the baseline throughput of the RDP. When the system detects that the effective throughput of the RDP is below 30% of the baseline throughput for two consecutive sampling periods, and the number of flow control triggers and retries is continuously increasing, it can be determined that the RDP has failed. The requirement for a continuous increase in the number of flow control triggers is to avoid misjudging a throughput decrease due to insufficient storage device processing capacity as a network failure. If only the throughput decreases, but the number of flow control triggers and retries does not change significantly, it may be due to insufficient storage device processing capacity, and failover is not required.

[0072] A2. When the number of retransmissions in the Transmission Control Protocol (TCP) plane exceeds 50 within 5 seconds, or the congestion window size remains at 1 and the round-trip delay is greater than 100 milliseconds, the TCP plane is considered to be faulty.

[0073] For the Transmission Control Protocol (TCP) plane, a retransmission count exceeding 50 within 5 seconds indicates a severe packet loss problem in the TCP connection. When the congestion window remains at 1 and the round-trip time (RTT) is greater than 100 milliseconds, the TCP connection has entered a state of severe congestion, making normal data transmission almost impossible. A congestion window of 1 means that the TCP can only send one message at a time and then wait for acknowledgment, resulting in extremely low throughput. A RTT greater than 100 milliseconds indicates very high network latency, possibly due to network congestion or link failure. Both of these situations indicate that the TCP plane can no longer provide normal service quality and requires failover.

[0074] Furthermore, the values ​​of the fault determination criteria can be adjusted according to the actual network environment and service requirements. For example, for services with extremely high reliability requirements, the fault determination threshold can be appropriately lowered to allow for timely switching in the event of minor faults. For services with extremely high continuity requirements, the fault determination threshold can be appropriately increased to avoid unnecessary switching due to temporary network fluctuations. This application does not impose any restrictions on these values. The system can also be combined with other fault detection mechanisms, such as network card link status detection and heartbeat detection, to improve the accuracy of fault determination.

[0075] Compared to existing technologies, which typically rely on application-layer timeout mechanisms to detect transport plane failures, the application layer assumes a transport plane failure when an I / O request sent by the application layer does not return within a preset timeout period and attempts to switch to another transport plane. This switching delay is usually on the order of seconds because the timeout period is typically set to several seconds or even tens of seconds to avoid false positives. During the switching delay, the service is completely interrupted. For services like AI training, which have extremely high continuity requirements, this interruption can cause the graphics processor to be idle for a long time, resulting in a huge waste of computing resources. In contrast, this application can detect transport plane failures within microseconds by collecting statistical indicators of the transport layer in real time, greatly shortening the fault detection time and gaining valuable time for rapid switching.

[0076] Optional, such as Figure 4 As shown, the step of binding newly arriving IO requests to the token bucket of the healthy plane for scheduling, while maintaining ongoing IO requests that have been admitted through the fault plane token bucket and issued before the fault occurred to continue execution on the fault plane path until completion, includes: S310 sets the atomic switching flag in the tenant structure and updates the current priority transport plane identifier to the healthy plane.

[0077] The atomic switch flag is an atomic variable that guarantees the atomicity of read and write operations in a multi-CPU core environment. An atomic operation is one that cannot be interrupted by the thread scheduling mechanism; once started, it runs continuously until completion without any context switching. Using the atomic switch flag ensures that during the switch, some new I / O operations are not routed to the faulty plane while others are routed to the healthy plane. Updating the atomic switch flag is a single-step atomic operation that does not require holding any locks, resulting in extremely fast execution speeds, completing within nanoseconds.

[0078] For example, the current priority transport plane flag can be an enumerated variable with values ​​for either the Remote Direct Memory Access plane or the Transport Control Protocol (TCP) plane. When the system detects a failure in the Remote Direct Memory Access plane, it atomically updates the value of the atomic switch flag to the TCP plane. After the update, all subsequent arriving I / O requests will read this new value and be routed to the TCP plane's token bucket for quota requests. After updating the atomic switch flag, the system executes a write memory barrier instruction to ensure that the update is visible to subsequent read operations on all CPU cores in the system. The write memory barrier instruction forces data in the CPU cache to be flushed to main memory, allowing other CPU cores to read the latest value.

[0079] S320, before a new IO request is admitted to the token bucket, the atomic switching flag is read, the corresponding token bucket is found according to the health plane indicated by the atomic switching flag, the quota is applied for, and the protocol type of the plane is marked.

[0080] S330: For ongoing IO requests that have been admitted through the fault plane token bucket and sent to the network protocol stack at the switching time, their original transmission paths are preserved. When the ongoing IO request is completed, the consumed quota is deducted from the fault plane token bucket through a statistical hook.

[0081] In this embodiment, for ongoing I / O requests that have already passed the fault plane token bucket admission and been issued to the network protocol stack at the time of handover, the system will not intervene in any way. These I / O requests will continue to be transmitted along the original transmission path until completion or failure. When these I / O requests are completed, the system will deduct the actual token amount consumed by these I / O requests from the fault plane token bucket through a statistical hook function registered in the block device layer. The statistical hook function is a callback mechanism provided by the block device layer. When an I / O request is completed, the system will call the registered hook function to pass relevant information about the I / O request. Through the statistical hook function, the system can accurately record the actual bandwidth consumed by each I / O request, thereby achieving accurate management of the token bucket. This natural exhaustion strategy can avoid data loss or inconsistency problems caused by forcibly interrupting ongoing I / O requests; at the same time, it does not require sending queue withdrawal instructions to the network card driver, greatly simplifying the handover process and reducing handover latency.

[0082] S340, the migration ends when all in-process IO requests on the fault plane are completed.

[0083] In this embodiment, the system maintains an in-transit IO tracking table to record information on all issued but not yet completed IO requests (i.e., ongoing IO requests, also referred to as in-transit IO requests). The in-transit IO tracking table can be a lightweight hash structure, using the IO identifier as the key, storing information such as the logical block address range, transport plane type, and commit time for each IO request. When an IO request is issued to the network protocol stack, the system adds its information to the in-transit IO tracking table; when an IO request is completed, the system deletes its record from the in-transit IO tracking table. When the in-transit IO tracking table of the fault plane is empty, it indicates that all ongoing IO requests on the fault plane have been completed, and the migration process officially ends. During the migration process, the system periodically checks the in-transit IO tracking table of the fault plane. When the tracking table is found to be empty, some cleanup operations are performed, such as restoring the parameters of the fault plane's token bucket and releasing temporarily allocated resources.

[0084] Compared to existing technologies, which typically forcibly cancel all ongoing I / O requests on the faulty plane during transport plane switching and then resend these I / O requests to the healthy plane, this approach leads to a large number of I / O request retransmissions, increasing network load and I / O request latency. Furthermore, forcibly canceling ongoing I / O requests can result in data loss or inconsistency, especially for write I / O requests. If some data has already been written to remote storage while other data is canceled, data incompleteness will occur. In contrast, this application employs a request-granularity-based path binding and natural exhaustion strategy, switching paths only for newly arriving I / O requests. Already issued ongoing I / O requests continue to execute on their original paths until completion. This approach does not require canceling or resending any ongoing I / O requests. Upper-layer applications will only perceive slight latency jitter in individual I / O requests without receiving any error returns, and the entire migration process is completely transparent to upper-layer applications.

[0085] In some embodiments, before binding a newly arriving IO request to the health plane, the method further includes: checking whether the logical block address range of the newly arriving IO request overlaps with the logical block address range of an IO request in progress on the fault plane; if there is an overlap, the distribution of the newly arriving IO request is suspended until the overlapping IO request in progress is completed before it is distributed to the health plane.

[0086] Logical block addresses are addresses used in storage devices to identify the location of data blocks, and each I / O request corresponds to a contiguous range of logical block addresses. If the logical block address range of a newly arriving I / O request overlaps with the logical block address range of an ongoing I / O request on a fault plane, both I / O requests will access the same data block. Allowing these two I / O requests to execute simultaneously on different transport planes can lead to data inconsistency. For example, if a write I / O request is writing data to logical block addresses 0 to 1023 via the remote direct memory access plane, while another write I / O request is writing different data to logical block addresses 512 to 1535 via the transport control protocol plane, the final data on the storage device may be a mixture of the two write I / O requests, making it impossible to determine which write I / O request's data will be retained. For read I / O requests, if a write I / O request is accessing the same data block simultaneously, the read I / O request may read some old data and some new data, resulting in data inconsistency.

[0087] For example, when a new I / O request arrives at the block device layer, the system first reads the atomic switching flag to determine the current priority transmission plane. Then, the system queries the in-transit I / O trace table to check if the logical block address range of the newly arrived I / O request overlaps with the logical block address range of any in-transit I / O request on the faulty plane. If there is no overlap, the newly arrived I / O request is directly dispatched to the token bucket of the healthy plane for quota application. If there is an overlap, the newly arrived I / O request is added to the waiting queue of the block device layer and temporarily suspended. Once the overlapping in-transit I / O requests complete and are removed from the in-transit I / O trace table, the system wakes up the newly arrived I / O request in the waiting queue and re-performs the logical block address overlap detection. If there is no overlap at this point, the newly arrived I / O request is dispatched to the healthy plane for processing.

[0088] Specifically, the method for detecting overlapping logical block addresses is as follows: For two logical block address ranges A and B, where A starts at startA and ends at endA, and B starts at startB and ends at endB, if startA <= endB and startB <= endA, then these two logical block address ranges overlap. The system iterates through all records in the in-transit IO tracking table of the fault plane, comparing each record with the logical block address range of a newly arriving IO request. If any record is found to overlap with the logical block address range of a newly arriving IO request, it is determined that an overlap exists. In this embodiment, the token bucket of the healthy plane is pre-expanded before migration, and the overlap of logical block addresses between newly arriving IO requests and in-process IO requests is detected, ensuring data consistency and smooth bandwidth transition during the switchover process. To improve the efficiency of overlap detection, the system can sort the records in the in-transit IO tracking table according to logical block addresses. This allows the use of a binary search method to quickly find whether there are overlapping records without traversing all records.

[0089] Furthermore, I / O requests in the waiting queue can be sorted by priority. Higher-priority I / O requests can be woken up and processed first. For example, read I / O requests typically have a higher priority than write I / O requests because read I / O requests are more latency-sensitive. The system can also set different priorities for different types of services, such as online inference services having a higher priority than offline training services. This application does not limit this; those skilled in the art can design suitable waiting queue scheduling algorithms based on actual business needs. When the waiting time of an I / O request in the waiting queue exceeds a preset timeout, the system can mark the I / O request as timed out and return an error to the upper-layer application. However, since the migration process is usually completed within sub-seconds, timeouts rarely occur.

[0090] Compared to existing technologies, which typically do not consider data consistency issues between ongoing I / O requests and newly arriving I / O requests during transport plane switching, allowing two I / O requests accessing the same data block to execute simultaneously on different transport planes can lead to data inconsistency and corruption. This application effectively avoids this by performing logical block address overlap detection during the migration process. When overlap is detected, the distribution of the newly arriving I / O request is suspended until the overlapping I / O request in transit completes. This mechanism ensures that the same data block is accessed only by I / O requests on one transport plane at a time, thus guaranteeing data consistency.

[0091] Based on the same inventive concept, this application also provides an RDMA / TCP dual-transmission plane cooperative QoS system, such as... Figure 5 As shown, it includes: The dual-plane token bucket isolation module 10 is used to establish four token buckets for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. The module configures a first time slice granularity and a first smoothing suppression strength for the remote direct memory access plane, and configures a second time slice granularity and a second smoothing suppression strength for the transmission control protocol plane. The first time slice granularity is greater than the second time slice granularity, and the first smoothing suppression strength is less than the second smoothing suppression strength. The transport layer feedback closed-loop module 20 is used to collect the number of flow control triggers, the number of explicit congestion notification markers, and the number of retries of the remote direct memory access plane, as well as the round-trip delay, the number of retransmissions, and the congestion window size of the transport control protocol plane, and adjust the filling rate and suppression parameters of the corresponding token bucket according to the collected data. The cross-plane migration module 30 is used to perform admission control on IO requests based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, the newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, the ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion.

[0092] The following section provides a detailed description of the specific implementation of this application, using a specific scenario of checkpoint saving in an artificial intelligence training cluster as an example.

[0093] Assume that the artificial intelligence training cluster contains multiple graphics processor computing nodes, each of which is simultaneously connected to a high-speed remote direct memory access network and a low-speed transmission control protocol network, and the remote storage node provides block storage services through both protocols.

[0094] When the RDMA / TCP dual-transmission plane collaborative QoS system of this application is applied to this scenario, the specific configuration information is as follows: The system maintains four independent token buckets for each tenant at the operating system kernel block device layer, corresponding to four traffic types: Remote Direct Memory Access (RDA) read, RDA write, Transmission Control Protocol (TCP) read, and TCP write. Each token bucket maintains three parameters: current available quota, bucket capacity limit, and fill rate per millisecond. Each of the four buckets is protected by an independent spinlock, allowing different CPU cores to simultaneously request quotas from different buckets without lock contention. The capacity limit for each token bucket can be set to 1000 token units, and the fill rate can be set to 250 token units per millisecond. Each token unit corresponds to a 4KB data block, therefore each token bucket can provide an average bandwidth of 1000MB per second. This application does not impose restrictions on these values.

[0095] The system also maintains a protocol-aware parameter scheduler, which internally stores a parameter mapping table. When a tenant's current preferred transport protocol is Remote Direct Memory Access, the time slice granularity is set to 5 milliseconds, and the smoothing suppression strength is set to 50. When the preferred transport protocol switches to Transmission Control Protocol (TCP), the time slice granularity automatically adjusts to 2 milliseconds, and the smoothing suppression strength adjusts to 80. The time slice granularity determines the minimum interval for token bucket filling operations, and the smoothing suppression strength determines the extent to which the token bucket tightens quota issuance when burst traffic is detected.

[0096] The system maintains two statistical acquisition structures at the transport layer. The Remote Direct Memory Access (RDA) statistics structure records the cumulative number of flow control triggers, the cumulative number of explicit congestion notifications, the cumulative number of retries, and the number of active queue pairs within the most recent second. The Transmission Control Protocol (TCP) statistics structure records the average round-trip time (RTT), RTT variance, cumulative number of retransmissions, current congestion window size, and number of active connections within the most recent sampling period. Statistical acquisition is performed by directly mapping and reading existing performance counters in the network interface card (NIC) driver or protocol stack, without introducing additional data copying overhead.

[0097] At the end of each sampling period, the system executes feedback closed-loop logic. For the Remote Direct Memory Access (RDA) plane, if the number of flow control triggers increases by more than ten times within one second, or the number of retries exceeds a preset threshold, the plane is considered congested. Actions include reducing the RDA bucket fill rate to 70% of its current value and increasing the smoothing suppression strength to 80%. For the Transmission Control Protocol (TCP) plane, if the average round-trip time exceeds twice the historical baseline and the number of retransmissions continues to increase, the plane is considered congested. Actions include reducing the TCP bucket fill rate to 60% of its current value. After the congestion indicators return to normal for three consecutive sampling periods, the fill rate gradually recovers to the baseline value in increments of 10% per period.

[0098] The cross-plane migrater maintains three fields in the tenant structure: the current priority transfer plane identifier, the atomic switching flag, and the in-transit IO trace table. The in-transit IO trace table is a lightweight hash structure that records the logical block address range of each issued but not yet completed IO using the IO identifier as the key, and is used for conflict detection within the migration window.

[0099] When a remote direct memory access network card failure occurs in the above checkpoint saving scenario, the complete operation process of the RDMA / TCP dual transport plane cooperative QoS system of this application is as follows: Initially, a training task triggers a checkpoint save operation after multiple iterations. The file system layer encapsulates the checkpoint data into multiple large-granularity sequential write I / O requests and sends them to the block device layer. Since the current priority transport plane is Remote Direct Memory Access (RDMI), these write I / O requests are routed to the RMI write token bucket for quota application during the token bucket lookup phase. The RMI write token bucket has sufficient available quota; after the I / O requests are admitted, they are marked as RMI protocol types, and the commit descriptor is registered in the send queue of the RMI network interface card (NIC). The NIC then writes the data directly to the memory buffer of the remote storage node via a high-speed link. Simultaneously, the transport layer statistics collection structure continuously updates various metrics of the RMI plane in the background.

[0100] Subsequently, the Remote Direct Memory Access (RDA) network interface card (NIC) on the graphics processor node began experiencing intermittent link jitter due to a hardware issue. Over the next period, the RDA statistics structure recorded a significant increase in retries and flow control triggers, while effective throughput dropped sharply. The transport layer feedback loop read these statistics at the next sampling period and found that both retries and flow control triggers had exceeded preset thresholds, and effective throughput had fallen to a low level of the baseline. The feedback loop determined that the RDA plane had experienced severe congestion that could not be automatically recovered from and transmitted the determination along with a migration trigger signal to the cross-plane migrater.

[0101] Upon receiving the trigger signal, the cross-plane migrater first performs a pre-migration operation on the target Transmission Control Protocol (TCP) plane. The migrater temporarily increases the maximum capacity of the TCP write token bucket from 1,000 tokens to 1,500 tokens, while simultaneously replenishing the current available quota of the TCP write token bucket to match the new maximum capacity. This ensures that the target plane will not experience queuing delays due to insufficient quota when receiving incoming traffic. This pre-expansion operation is completed under the spinlock protection of the TCP write token bucket, and the entire process takes microseconds.

[0102] The migrater writes the protocol identifier value of the transport control plane to the atomic switching flag field of the tenant structure, and then executes a write memory barrier instruction to ensure that the update is visible to subsequent read operations on all CPU cores in the system. This write operation is the only global control variable that needs to be modified during the migration process. The migrater does not traverse any I / O request queues, does not modify the data structures of any ongoing I / O requests, and does not send any queue withdrawal instructions to the network card driver.

[0103] In the next IO request dispatch cycle after the atomic switch flag takes effect, the checkpoint save operation continues to generate new sequential write IO requests at the file system layer. Upon reaching the block device layer, these newly arriving IO requests first read the atomic switch flag during the token bucket admission decision. The flag indicates that the current priority transport plane has changed to Transmission Control Protocol (TCP). Therefore, these IO requests no longer search the Remote Direct Memory Access (RDMI) write token bucket, but are directly routed to the TRP write token bucket to request write quota. Since the TRP write token bucket has been expanded to 1500 token units and has sufficient quota during the pre-migration phase, all these IO requests are directly admitted. Admitted IO requests are marked as TRP protocol types, and the commit descriptor enters the TRP protocol stack processing flow. The TRP protocol stack establishes a socket send context for each IO request and sends the contents of the data buffer to the remote storage node via the Ethernet link in the form of TRP messages. The remote storage node's TRP listening port receives these messages and writes the data to the corresponding storage medium.

[0104] At the moment the atomic switch flag takes effect, multiple large-granularity sequential write I / O requests have been admitted through the Remote Direct Memory Access (RDM) write token bucket and issued to the send queue of the RDM network interface card (NIC). The migrater does not interfere with the execution of these I / O requests, and the RDM NIC continues to complete the transmission and reception of the remaining data according to the original RDM semantics. After each I / O request completes the RDM write of all data segments and receives the memory write confirmation from the remote end, the NIC generates a completion event in the RDM completion queue, the interrupt handler calls the block-level completion callback, the completion callback releases the data buffer occupied by the I / O request, deducts the number of bytes transferred from the current available quota of the RDM write token bucket through a statistical hook, and deletes the record of the I / O request from the in-transit I / O trace table. When the last remaining I / O request completes and is deleted from the in-transit I / O trace table, the number of in-transit requests in the RDM write token bucket drops to zero, and the natural exhaustion process ends.

[0105] During the parallel execution of new I / O request distribution and the natural exhaustion of legacy I / O requests, the block device layer queries the in-transit I / O trace table before distributing a new I / O request to the Transmission Control Protocol (TCP) plane. The query checks if the logical block address range of the new I / O request overlaps with any in-transit I / O requests from the Remote Direct Memory Access (RDA) plane. If an overlap is detected, the new I / O request is not immediately sent to the TCP plane but is temporarily suspended in the block device layer's waiting queue. Once the overlapping legacy I / O request completes and is removed from the in-transit I / O trace table, the new I / O request in the waiting queue is awakened, and the logical block address conflict detection is re-executed. Since there is no overlap now, the new I / O request is allowed and enters the TCP plane's distribution process. This conflict detection mechanism ensures that the same data block is not concurrently written by I / O requests from both the RDA plane and the TCP plane, preventing data inconsistency on the remote storage medium.

[0106] After migration, the cross-plane migrater continuously collects transport layer statistics of the remote direct memory access plane at fixed intervals. Once the maintenance personnel repaired the faulty remote direct memory access network interface card (NIC), the physical quality of the remote direct memory access link returned to normal. In the following consecutive sampling periods, all indicators of the remote direct memory access plane returned to normal levels. The cross-plane migrater determined that the remote direct memory access plane had met health conditions for several consecutive sampling periods and then initiated the reverse migration process. The reverse migration steps are symmetrical to the forward migration. First, the capacity of the remote direct memory access write token bucket is temporarily increased to 1.5 times the baseline value and replenished. Then, the atomic switch flag is written back to the protocol identifier value of the remote direct memory access plane. New IO requests re-enter the remote direct memory access path, and the legacy IO requests of the transmission control protocol plane are naturally exhausted. The remaining quota in the transmission control protocol write token bucket at the time of reverse migration is proportionally reduced and added to the remote direct memory access write token bucket to ensure that the reverse migration does not cause an initial quota shortage in the remote direct memory access plane. The system state after the reverse migration is completely consistent with that before the fault occurred.

[0107] During the aforementioned forward and reverse migration processes, the protocol-aware parameter scheduler automatically completes the parameter switching within the first scheduling cycle after each atomic switching flag update. When the priority transmission plane changes from Remote Direct Memory Access (RDA) to Transmission Control Protocol (TCP), the time slice granularity is adjusted from five milliseconds to two milliseconds, and the smoothing suppression strength is adjusted from fifty to eighty. This ensures that the token bucket filling behavior of the TCP plane matches the congestion-sensitive characteristics of the protocol, preventing sudden traffic surges in the TCP from causing severe oscillations in its own congestion window. When a reverse migration occurs and the priority transmission plane reverts to RDA, the parameter scheduler restores the time slice granularity and smoothing suppression strength to five milliseconds and fifty, respectively, to match the high throughput and low latency characteristics of RDA.

[0108] Through the coordinated operation of the above steps, during the hardware failure of the remote direct memory access network card, the application layer only perceived a slight I / O request latency jitter during the entire checkpoint saving operation, without receiving any I / O request error returns, and the training task was not interrupted. The switching latency from fault detection to the start of new I / O requests along the transmission control protocol plane path during the migration process was in the sub-millisecond range, and there was no precipitous drop in bandwidth utilization during the migration process. The reverse migration after fault recovery was equally smooth and transparent.

[0109] In this application's technical solution, the design of four independent token buckets solves the problem of bandwidth encroachment between protocols. Existing technologies using a single token bucket design cause large-granularity I / O requests for remote direct memory access to be frequently interrupted by small-granularity I / O requests from the transmission control protocol, leading to a decrease in effective throughput. This application, however, completely isolates the four traffic types, ensuring that each type has an independent bandwidth quota and does not interfere with the others. Simultaneously, it configures differentiated time-slice granularity and smoothing suppression strength for different protocols, matching the token bucket behavior to the transmission characteristics of the protocols and fully leveraging the advantages of each protocol.

[0110] The design of a feedback loop at the transport layer solves the problem of double suppression. In existing technologies, there is a lack of information pathways between the block layer and the transport layer. When network congestion occurs, the transport layer actively reduces its speed, but the block layer continues to fill tokens at a fixed rate, resulting in double suppression. This application, however, dynamically adjusts the token bucket filling rate by collecting statistical indicators from the transport layer in real time, ensuring that the block layer's rate control and the transport layer's adaptive speed reduction are aligned, thereby eliminating double suppression and improving effective throughput in congested scenarios.

[0111] The cross-plane migration design based on atomic switching flags and a natural exhaustion strategy solves the problems of high failover latency and data inconsistency. Existing technologies rely on application-layer timeout retries for switching, resulting in latency on the order of seconds, and the forced cancellation of ongoing I / O requests leads to data inconsistency. This application, however, uses transport layer statistical metrics for fault detection, enabling fault detection within microseconds. It achieves path switching for new I / O requests through atomic switching flags and handles ongoing I / O through a natural exhaustion strategy. The entire migration process is completely transparent to upper-layer applications, and data consistency is guaranteed through logical block address overlap detection.

[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A collaborative QoS scheduling method for RDMA / TCP dual transport planes, characterized in that, include: Four token buckets are established for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. A first time slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time slice granularity and a second smoothing suppression strength are configured for the transmission control protocol plane. The first time slice granularity is larger than the second time slice granularity, and the first smoothing suppression strength is smaller than the second smoothing suppression strength. Collect the flow control trigger count, explicit congestion notification flag count, and retry count of the remote direct memory access plane, as well as the round-trip time, retransmission count, and congestion window size of the transmission control protocol plane, and adjust the corresponding token bucket filling rate and suppression parameters based on the collected data; Admission control for IO requests is performed based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion. Adjust the filling rate and suppression parameters of the corresponding token bucket based on the collected data, including: When the number of flow control triggers in the remote direct memory access plane increases beyond a preset threshold or the number of retries exceeds a preset retry threshold within 1 second, the filling rate of the remote direct memory access token bucket is reduced and its smoothing suppression strength is increased. The remote direct memory access token bucket includes a remote direct memory access read token bucket and a remote direct memory access write token bucket. When the average round-trip time of the Transmission Control Protocol (TCP) plane exceeds a preset multiple of the historical baseline value and the number of retransmissions continues to increase, the filling rate of the TCP token bucket is reduced. The TCP token bucket includes a TCP read token bucket and a TCP write token bucket.

2. The method according to claim 1, characterized in that, Create four token buckets for the same tenant, including: In the tenant data structure, separate token bucket structures are allocated for remote direct memory access read, remote direct memory access write, transmission control protocol read, and transmission control protocol write. Each token bucket structure contains a capacity limit field, a current available quota field, and a fill rate field, and concurrent access is protected by an independent spin lock.

3. The method according to claim 1, characterized in that, Configure a first time-slice granularity and a first smoothing suppression strength for the remote direct memory access plane, and configure a second time-slice granularity and a second smoothing suppression strength for the transport control protocol plane, including: Set the first time slice granularity of the remote direct memory access plane to 5 milliseconds and the first smoothing suppression strength to 50. Set the second time slice granularity of the transmission control protocol plane to 2 milliseconds and the second smoothing suppression strength to 80.

4. The method according to claim 1, characterized in that, The data collection includes the number of flow control triggers, explicit congestion notification flags, and retries for the remote direct memory access plane, as well as the round-trip time, retransmission count, and congestion window size for the transmission control protocol plane, including: The number of flow control triggers, the number of explicit congestion notifications, and the number of retries in the remote direct memory access plane are read using the performance counters in the network card driver. The round-trip time, retransmission count, and congestion window size of the Transmission Control Protocol (TCP) plane are read from the TCP stack. The round-trip time includes the average round-trip time and the round-trip time variance.

5. The method according to claim 1, characterized in that, Reduce the fill rate of the remote direct memory access token bucket, including adjusting the fill rates of the remote direct memory access read token bucket and the remote direct memory access write token bucket to 70% of their current values; The improvement of its smoothing suppression strength includes: increasing the smoothing suppression strength to 80.

6. The method according to claim 1, characterized in that, A fault was detected in the remote direct memory access plane or the transport control protocol plane, including: When the number of retries in the remote direct memory access plane exceeds 100 within 5 seconds, or when the number of flow control triggers continues to rise and the effective throughput drops to less than 30% of the baseline throughput, the remote direct memory access plane is considered to be faulty. A transmission control protocol plane (TCP) failure is determined when the number of retransmissions exceeds 50 within 5 seconds, or when the congestion window size remains at 1 and the round-trip delay is greater than 100 milliseconds.

7. The method according to claim 1, characterized in that, Newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. Meanwhile, ongoing IO requests that have already been admitted through the fault plane token bucket and issued before the fault occurred continue to execute on the fault plane path until completion, including: Set an atomic switching flag in the tenant structure and update the current priority transport plane identifier to the healthy plane; Before a new IO request is admitted to the token bucket, the atomic switching flag is read, the corresponding token bucket is found according to the health plane indicated by the atomic switching flag, the quota is applied for, and the protocol type of the plane is marked. For ongoing I / O requests that have been admitted through the fault plane token bucket and have been sent to the network protocol stack at the switching time, their original transmission path is retained. When the ongoing I / O request is completed, the consumed quota is deducted from the fault plane token bucket through a statistical hook. The migration ends when all in-process I / O requests on the fault plane are completed.

8. The method according to claim 7, characterized in that, Before binding newly arriving IO requests to the health plane, the following steps are also included: Check if the logical block address range of the newly arrived IO request overlaps with the logical block address range of the IO request in progress on the fault plane. If there is an overlap, postpone the distribution of the newly arrived IO request until the overlapping IO request in progress is completed before distributing it to the healthy plane.

9. A dual-transmission plane collaborative QoS scheduling system for RDMA / TCP, characterized in that, include: A dual-plane token bucket isolation module is used to establish four token buckets for the same tenant at the operating system kernel block device layer. The four token buckets include a remote direct memory access read token bucket, a remote direct memory access write token bucket, a transmission control protocol read token bucket, and a transmission control protocol write token bucket. A first time slice granularity and a first smoothing suppression strength are configured for the remote direct memory access plane, and a second time slice granularity and a second smoothing suppression strength are configured for the transmission control protocol plane. The first time slice granularity is greater than the second time slice granularity, and the first smoothing suppression strength is less than the second smoothing suppression strength. The transport layer feedback closed-loop module is used to collect the number of flow control triggers, the number of explicit congestion notification markers, and the number of retries of the remote direct memory access plane, as well as the round-trip delay, the number of retransmissions, and the congestion window size of the transport control protocol plane, and adjust the filling rate and suppression parameters of the corresponding token bucket according to the collected data. Adjust the filling rate and suppression parameters of the corresponding token bucket based on the collected data, including: When the number of flow control triggers in the remote direct memory access plane increases beyond a preset threshold or the number of retries exceeds a preset retry threshold within 1 second, the filling rate of the remote direct memory access token bucket is reduced and its smoothing suppression strength is increased. The remote direct memory access token bucket includes a remote direct memory access read token bucket and a remote direct memory access write token bucket. When the average round-trip time of the Transmission Control Protocol (TCP) plane exceeds a preset multiple of the historical baseline value and the number of retransmissions continues to increase, the filling rate of the TCP token bucket is reduced. The TCP token bucket includes a TCP read token bucket and a TCP write token bucket. The cross-plane migration module is used to perform admission control on IO requests based on the adjusted token bucket. When a failure is detected in the remote direct memory access plane or the transmission control protocol plane, the newly arriving IO requests are bound to the token bucket of the healthy plane for scheduling. At the same time, ongoing IO requests that have been admitted through the token bucket of the failed plane before the failure occurred continue to be executed on the path of the failed plane until completion.

Images