Operating method of perf, and device therefor

Abstract

The present disclosure relates to an operating method of a PeRF, comprising: an on-loading process operation which is executed in a main thread to manage a lightweight task and triggers an isolation process; and an off-loading process operation which is executed in a background thread to perform a complex task, wherein the off-loading process operation can include the operations of: using a PeRF master to coordinate one or more PeRF workers; and executing the one or more PeRF workers according to the isolation process.

Description

PERF operation method and device thereof

[0001] The following disclosure relates to the operating method and device of PeRF.

[0002] Remote Direct Memory Access (RDMA) can be a technology that minimizes CPU intervention when transferring data over a network, delivering high throughput and low latency. RDMA enables direct memory access from the network interface card (NIC), maximizing data transfer speeds and reducing CPU utilization. RDMA can offer zero-copy data transfers, ultra-low latency, high throughput, and low CPU utilization. RDMA can play a crucial role in high-performance computing, databases, big data analytics, machine learning, and distributed storage systems.

[0003] However, RDMA has primarily been used in single-tenant environments, and multi-tenant cloud environments may require addressing performance isolation, security, and scalability issues. Existing hardware- and software-based performance isolation solutions may not effectively respond to dynamic resource changes or may cause performance degradation. For these reasons, new solutions may be needed to efficiently manage RDMA resources and achieve performance isolation in multi-tenant environments.

[0004] A PeRF operating method according to one embodiment includes an onloading process operation that runs in a main thread to manage lightweight tasks and trigger an isolation process, and an offloading process operation that runs in a background thread to perform complex tasks, wherein the offloading process operation may include an operation of coordinating one or more PeRF workers using a PeRF master, and an operation of executing the one or more PeRF workers according to the isolation process.

[0005] The above onloading process operation may include an operation of intercepting an application's verb call, an operation of estimating an average message size through a message size estimator, an operation of estimating the number of QPs through a QP (Queue Pair) number estimator, an operation of enforcing a network policy specified by a system administrator through a network policy manager, and an operation of dynamically triggering the isolation process depending on the type of application through an isolation manager.

[0006] The above isolation manager monitors the distribution and network status of global tenants, and can selectively send a signal to the offloading process to activate the isolation process based on detected performance abnormalities.

[0007] The PeRF master can monitor the global state of the tenant and create and schedule the PeRF worker according to the isolation process.

[0008] The PeRF worker may include an application classifier that determines the application type, a task maintenance scheduler that manages packet transmission, and dynamically performs the preemption process.

[0009] The above application classifier can classify the type of the application based on the estimated message size and QP number.

[0010] The above task maintenance scheduler may include at least one of a Large Message Scheduling Engine (LMSE), a Multi-QP Scheduling Engine (MQSE), and an Early Completion Engine (ECE).

[0011] The above large message scheduling engine can interrupt large message transmission to allocate resources for small message transmission.

[0012] The above multi-queue pair scheduling engine can manage the number of active queue pairs of a multi-queue pair application using PAUSE and RESUME operations.

[0013] The above early completion engine may poll additional WRs generated by the preemption mechanism to prevent saturation of the transmission queue.

[0014] An electronic device according to one embodiment includes a memory storing instructions and one or more processors, wherein the instructions, when executed by the one or more processors, cause the electronic device to perform an onloading process that runs in a main thread to manage lightweight tasks and trigger an isolation process, and an offloading process that runs in a background thread to perform complex tasks, wherein the offloading process may be a process that coordinates one or more PeRF workers using a PeRF master and executes the one or more PeRF workers according to the isolation process.

[0015] The above onloading process may be a process that intercepts the verb call of the application, estimates the average message size through a message size estimator, estimates the number of QPs through a QP (Queue Pair) number estimator, enforces a network policy specified by a system administrator through a network policy manager, and dynamically triggers the isolation process according to the type of application through an isolation manager.

[0016] The above isolation manager monitors the distribution and network status of global tenants, and can selectively send a signal to the offloading process to activate the isolation process based on detected performance abnormalities.

[0017] The PeRF master can monitor the global state of the tenant and create and schedule the PeRF worker according to the isolation process.

[0018] The above PeRF worker is an application classifier that determines the application type;

[0019] It may include a task maintenance scheduler that manages packet transmission and dynamically performs preemption processes.

[0020] The above application classifier can classify the type of the application based on the estimated message size and QP number.

[0021] The above task maintenance scheduler may include at least one of a Large Message Scheduling Engine (LMSE), a Multi-QP Scheduling Engine (MQSE), and an Early Completion Engine (ECE).

[0022] Figures 1a to 1c schematically illustrate data obtained by analyzing RNIC at a micro scale according to one embodiment.

[0023] FIGS. 2A to 3C are diagrams for explaining an abnormal phenomenon of RDMA in a multi-tenant environment according to one embodiment.

[0024] FIG. 4 is a block diagram schematically illustrating the structure and flow of PeRF to explain PeRF according to one embodiment.

[0025] FIG. 5 is a diagram for explaining a preemption mechanism of an RNIC in PeRF according to one embodiment.

[0026] FIG. 6a and FIG. 6b are diagrams for explaining MQSE operation and LMSE operation according to one embodiment.

[0027] FIG. 7 is a block diagram illustrating an electronic device according to one embodiment.

[0028] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0029] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0030] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0031] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0032] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0034] The embodiments can be implemented in various forms of products, such as personal computers, laptop computers, tablet computers, smartphones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. Hereinafter, the embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0035]

[0036] Remote Direct Memory Access (RDMA) can be a promising network technology that delivers high throughput and ultra-low latency with minimal CPU usage through zero-copy read / write operations in an application's virtual memory space. Various data-intensive applications, such as big data analytics, machine learning, distributed storage, and key-value stores, can benefit from RDMA's performance. However, because RDMA and its applications are typically designed for single-tenant use, deploying RDMA in multi-tenant cloud environments can be challenging. Consequently, multi-tenant RDMA deployments can face various challenges, including security, scalability, and performance isolation.

[0037] Without proper performance isolation in RDMA, some tenants may not achieve the expected performance, defeating the purpose of using RDMA. Several approaches exist to provide RDMA performance isolation in multi-tenant environments. These approaches can utilize hardware (HW) or software (SW) technologies to adjust data transfer rates for each tenant, thereby ensuring efficient resource utilization. Several RDMA virtualization frameworks have been proposed that leverage SR-IOV or virtual lanes (VLs) supported by RDMA hardware, such as switches or RNICs. While hardware-based frameworks can provide strict performance isolation, they may not adequately handle dynamic changes in tenant requirements.

[0038] In contrast, software-based solutions can focus on intercepting data transfer requests generated by applications on the Remote Direct Memory Access Network Interface Card (RNIC) and controlling each tenant's request rate from user space. Compared to hardware-based approaches, software-based approaches can achieve more flexible performance isolation from dynamic changes in network resource sharing. However, because they rely on reservation-based resource allocation to throttle transfer requests without hardware assistance, they can incur performance degradation.

[0039] In RDMA, the entire network stack is offloaded to the RNIC, and applications can interact with the RNIC using user-level RDMA APIs (IB Verbs). To initiate data communication, an application can create a Queue Pair (QP) and a Completion Queue (CQ). A QP can contain a Send Queue (SQ) and a Receive Queue (RQ). To send or receive a message, an application can post a Work Request (WR) to its QP (SQ or RQ) and ring the RNIC's doorbell, prompting the RNIC to retrieve the WR. This process completely offloads message transmission to the RNIC, freeing the application to perform other tasks. Upon completion of the message transmission, the RNIC pushes a Completion Queue Element (CQE) to the CQ, and the application can poll the CQE from the CQ to determine the completion of the message transmission.

[0040] RDMA can support two types of primitive operations: one-sided primitives (READ and WRITE operations) and two-sided primitives (SEND / RECV operations). All primitive operations are driven by the local application, and remote applications can operate passively.

[0041] When a local application issues a WR for a one-way primitive operation, the local RNIC can send a message containing the remote virtual address and memory key (data for a WRITE or a request for a READ) to the remote RNIC. This allows the remote RNIC to transfer data directly to remote memory space without involving the remote CPU.

[0042] For bidirectional primitive operations, when a local application posts a SEND WR to the local RNIC, the local RNIC can perform a SEND operation that transmits a message without any information about the remote memory space. The remote RNIC can process the received message by referencing the RECV WR previously posted by the remote application to the RQ.

[0043]

[0044] Figures 1a to 1c schematically illustrate data obtained by analyzing RNIC at a micro scale according to one embodiment.

[0045] Referring to Figures 1a to 1c, we can see the results of a fine-scale benchmark performed to understand how RNIC handles heterogeneous WRs (work requests) posted in parallel to multiple QPs.

[0046] In Figure 1a, the first benchmark test demonstrates how the RNIC schedules multiple QPs. In this first benchmark, each application can send message batches of different sizes using its own single QP. Referring to graph (110), we can see that when five applications send small messages (e.g., less than 1 KB), the RNIC achieves its maximum message delivery rate for a single QP. However, adding another application sending larger messages (1, 2, or 4 KB) can limit the message delivery rate of the other applications. This demonstrates that the RNIC uses a round-robin QP scheduling scheme that fairly handles WRs across multiple QPs, regardless of message size.

[0047] In Figures 1b and 1c, the second benchmark test may be to verify the packet scheduling of the RNIC when transmitting messages larger than the MTU (Maximum Transmission Unit). Two applications share the RNIC, each of which can use a single QP. One application batches and transmits messages ranging in size from 64 B to 1 GB as background traffic, while the other generates and transmits 16 B messages, measuring the message rate (see graph (120)) and latency (see graph (130)). To observe the effect of the MTU, the MTU can be varied from 1 KB to 4 KB. As the background message size increases from 64 B to the MTU, the message rate and latency gradually worsen, stabilizing when the message size reaches the MTU. This can occur because the RNIC processes all packets with the same priority, regardless of size. This can cause small packets to wait for the transmission of larger packets from the background traffic, which can cause delays in the small packets. Conversely, if the background message size exceeds the MTU, the performance degradation rate can stabilize. This is because when the RNIC encounters a packet larger than the MTU, it fragments it into MTU-sized packets for processing, ensuring that the delay experienced by smaller packets does not exceed the transmission time of the MTU-sized packet.

[0048] In conclusion, Figures 1a through 1c demonstrate two characteristics of the RNIC during data transmission. First, the RNIC processes WRs from multiple QPs using round-robin scheduling. Second, the RNIC evenly schedules packets of various sizes based on the MTU. Therefore, a method that reflects these RNIC characteristics and enables optimized performance isolation in situations where the RNIC is shared among multiple tenants may be needed.

[0049]

[0050] FIGS. 2A to 3C are diagrams for explaining an abnormal phenomenon of RDMA in a multi-tenant environment according to one embodiment.

[0051] Referring to Figures 2a to 3c, anomalies can be observed in three types of RDMA applications, depending on message size and speed. In a multi-tenant cloud environment, cloud administrators may need to appropriately manage system resources, including processing power, storage, and communication. However, RDMA networks can be difficult for administrators to effectively manage due to kernel bypass capabilities. RDMA's packet scheduling mechanism can cause unfair packet transmission opportunities between tenants, resulting in performance anomalies and resource exhaustion. Figures 2a to 3c analyze these anomalies through microscale experiments.

[0052] Figures 2a to 3c may be experimental results that classify three types of RDMA applications based on message size and speed. The three types may be delay-sensitive applications, message-intensive applications, and bandwidth-intensive applications.

[0053] Delay-sensitive applications (D_Apps) may generate small messages (e.g., significantly smaller than the MTU) infrequently, requiring short completion times. Message-intensive applications (M_Apps) may generate small messages in batches to achieve a high message rate. Bandwidth-intensive applications (B_Apps) may transmit large messages (e.g., larger than the MTU) to achieve high throughput. Each type of application can be further subdivided into two subtypes, depending on whether it utilizes a single QP or multiple QPs. In summary, there are six types of applications (B_Apps). single , B_App multi , M_App single , M_App multi, D_App single , D_App multi ) can be used to check for RDMA abnormalities.

[0054] Referring to Figures 2a and 2b, we can see the impact of an application using large messages on the transmission of small messages. As can be seen in Figure 2a, M_App single This B_App single When sharing RNIC with M_App single The message rate of B_App is greatly reduced, while single It can be seen that B_App achieves high throughput (94.77-96.89 Gbps) for all tasks. Similarly, in Fig. 2b, B_App single Silver D_App single We can see that the average and 99th percentile (99%) message completion times (MCT) increase while impairing the performance of B_App. On the other hand, single You can see that it achieves 97.94 Gbps. M_App single and D_App single When sharing this RNIC, we can see that no performance abnormalities occur because the message sizes are similar. In addition, two B_Apps generate messages of different sizes. single You can also see that performance abnormalities occur between them.

[0055] Referring to Figures 3a to 3c, performance anomalies can be observed depending on the number of QPs. The QP-level round-robin scheduling of the RNIC can cause anomalies in fair task processing by providing more packet transmission opportunities to multiple QP applications. In Figure 3a, B_App single and B_App multi You can see the comparison of throughput when competing for resources on the same RNIC. Accordingly, B_App multiAs the number of QPs in B_App increases single You can see that the processing capacity of M_App is decreasing. single and D_App single This B_App multi We can also see performance degradation when sharing RNICs.

[0056] M_App multi Do D_App single and M_App single This can degrade the processing performance of the RNIC. The RNIC has multiple processing units (PUs), each of which can independently process WRs posted to a QP. This allows the RNIC to transmit packets from multiple QPs in parallel and achieve the maximum message rate. For example, the RNIC achieves approximately 105 Mmps with nine QPs, while achieving a maximum of approximately 13 Mmps with a single QP.

[0057] In Figures 3b and 3c, M_App multi We can see that the message rate increases as the number of QPs increases. However, when the RNIC reaches its maximum capacity, M_App multi is M_App single and D_App single may begin to degrade its performance.

[0058]

[0059] Hereinafter, the Preemption-enabled RDMA Framework (PeRF), an RDMA framework with software-based preemption capabilities that provides performance isolation in a multi-tenant environment while maintaining the basic performance of RDMA, will be described in the following embodiments. Existing software-based solutions may attempt to adjust the transmission request rates of tenants to control packet scheduling of the RNIC. However, existing methods may have limitations because accurate estimation of network resources may be essential, which may lead to performance degradation. In other words, underestimating available resources may result in insufficient requests being delivered to the RNIC, which may degrade throughput and processing speed. Conversely, overestimating available resources may result in performance isolation not being achieved due to accumulated requests in the RNIC.

[0060] PeRF's goal may be to control packet scheduling on the RNIC without requiring accurate estimation of network resources. Unlike reservation-based, non-work-maintaining scheduling approaches that require accurate estimation of network resources, PeRF utilizes a preemption-based, work-maintaining scheduling approach to maximize network utilization while providing performance isolation. This can be achieved by utilizing specific commands provided by the RDMA user-level API (e.g., IB_WR_WAIT and IB_WR_ENABLE), which can pause packet processing for one connection while allowing other active connections to obtain processing opportunities. PeRF can interleave these preemptions between tenant-generated requests, allowing for flexible control of packet transmission on the RNIC. Furthermore, PeRF can selectively preempt applications with large messages or multiple connections, thereby providing additional processing opportunities for applications with small messages or a single connection.

[0061] Through task-maintaining scheduling and a novel preemption mechanism, PeRF can effectively isolate each tenant's RDMA resource utilization to dynamically adapt to changing network and tenant conditions, while maintaining high performance approaching hardware-based solutions. Furthermore, PeRF can be implemented using existing RDMA APIs.

[0062]

[0063] FIG. 4 is a block diagram schematically illustrating the structure and flow of PeRF to explain PeRF according to one embodiment.

[0064] Referring to Figure 4, PeRF (400) can be a software-based performance isolation framework that adapts to dynamic network conditions and diverse tenant requirements without compromising resource utilization. The aforementioned analysis of RDMA's multi-tenant environment confirms the need to prioritize the performance of applications using small messages or a single QP over applications using large messages or multiple QPs. Accordingly, PeRF (400) can operate according to three rules.

[0065] Rule 1 might be to isolate small message transmissions from large message transmissions. Rule 2 might be to isolate applications using a single QP from bandwidth-intensive or message-intensive applications using multiple QPs. Rule 3 might be to ensure that message transmissions for delay-sensitive applications are not interrupted.

[0066] Rules 1 and 2 enable PeRF (400) to prevent the aforementioned performance anomalies. To implement Rules 1 and 2, PeRF (400) can implement a preemption mechanism for task-maintaining performance isolation. Rule 3 enables PeRF (400) to perform isolated processing to maintain low latency for delay-sensitive applications.

[0067] PeRF (400) is implemented as a user-level library, and PeRF (400) can operate by dividing the process into two parts: an onloading (410) process and an offloading (420) process.

[0068] The onloading (410) process runs on the main thread, manages lightweight tasks, and can trigger isolated processes. The offloading (420) process runs on a background thread, capable of performing complex tasks.

[0069] In PeRF (400), the onloading (410) process can operate through a message size estimator (411), a QP number estimator (412), a network policy manager (413), and an isolation manager (414). First, the onloading (410) process can intercept verb calls of an application (e.g., RDMA command calls). Then, the message size estimator (411) can estimate the average message size, and the QP number estimator (412) can estimate the number of generated QPs. The estimated message size and QP number are transmitted to the network policy manager (413), and the network policy manager (413) can enforce a network policy (request rate limit or QP creation limit for each tenant) specified by the system administrator. In addition, the isolation manager (414) can dynamically trigger an isolation process depending on the application type based on the estimated message size. The isolation manager (414) monitors the distribution and network status of global tenants and can selectively send a signal to activate the isolation process in the offloading (420) process based on detected performance abnormalities.

[0070] An isolation manager (414) according to one embodiment is a D_App and an M_App singleThis can be implemented to allow the generated WR to be passed directly to the RNIC, bypassing the isolation process. M_App multi In this case, the isolation manager (414) can post the WR to an available QP or pass it to the offloading (420) process part to be posted later. All WRs of B_Appsingle / multi can be passed to the offloading (420) process part.

[0071] For applications that transmit both large and small messages, PeRF (400) may allow the QP to bypass the isolation process for small message requests, regardless of the type of application, if no large message transmission is in progress. App multi If one of the QPs has a high message rate similar to that of the M_App, PeRF (400) can apply the isolation process of the M_App to that QP. This behavior allows PeRF (400) to maintain low latency for delay-sensitive messages (e.g., may be D_App or other messages) while providing performance isolation to other RDMA applications.

[0072] The offloading (420) process can proceed according to the isolation process activation signal of the isolation manager (414).

[0073] The offloading (420) process can be implemented with two types of background threads, such as a PeRF master (421) and a PeRF worker. In PeRF (400), the offloading (420) process can be implemented with a PeRF master (421) and one or more PeRF workers (e.g., PeRF worker 1 to PeRF worker n (422-1 to 422-n)). The PeRF master (421) can coordinate one or more PeRF workers. In the offloading (420) process, when a signal to activate an isolation process is received by the isolation manager (414) of the onloading (410) process, PeRF (400) can execute one or more PeRF workers according to the isolation process.

[0074] In one embodiment, the PeRF master (421) can monitor the tenant's global state and coordinate multiple PeRF workers to ensure optimal CPU utilization. Each time a tenant launches a new RDMA application, the PeRF master (400) can create and schedule a new PeRF worker to manage the complex isolation process. That is, the PeRF master (421) can monitor the tenant's global state and create and schedule PeRF workers according to the isolation process.

[0075] PeRF workers share memory space with the main thread and can freely post WRs to the application's QP for isolation processes. Performance isolation of PeRF (400) can begin with identifying the requirements of applications running on multiple tenants. In the offloading (420) process, each of one or more PeRF workers may include an application classifier (423) that determines the application type and a work-conserving scheduler (424) that manages packet transmission and dynamically performs the preemption process.

[0076] An application classifier (423) according to one embodiment can classify an application type based on the estimated message size and QP count received from the onloading (410) process part. The application classifier (423) can intercept user-level RDMA commands (ibv_create_qp and ibv_post_send) and determine the application type based on the average message size estimated by the message estimator of the onloading (410) part and the number of generated QPs estimated by the QP count estimator (412). For example, if the average message size is 1 KB or more, the application can be classified as a B_App. For applications that use smaller messages, the application can potentially be classified as a D_App or an M_App. In addition, the application classifier (423) can include an additional classification step. For example, the number of unfinished WRs within the SQ (SQ_Len) can be monitored at 5 ms intervals, and if the maximum SQ_Len within the last 1,000 ms is below a certain threshold (T_h), the application classifier (423) can determine the application type based on the average message size estimated by the message estimator of the onloading (410) part and the number of generated QPs estimated by the QP count estimator (412). For example, if the average message size is 1 KB or more, the application can be classified as a B_App. For applications that use smaller messages, the application can potentially be classified as a D_App or an M_App. In addition, the application classifier (423) can include an additional classification step. For example, the number of unfinished WRs within the SQ (SQ_Len) can be monitored at 5 ms intervals, and if the maximum SQ_Len is less than a certain threshold (T_h) within the last 1,000 ms, the application classifier (423) can determine the application type based on the average message size estimated by the message estimator of the onloading (410) part and the number of generated QP M_App ) the application may be classified as M_App.

[0077] When an application utilizes multiple QPs, scheduling messages of different sizes and speeds across different QPs can increase the complexity of classifying application types for the application classifier (423). In this case, the application classifier (423) can classify applications based on the largest message size or fastest speed among all QPs.

[0078] A task maintenance scheduler (424) according to one embodiment can dynamically perform a preemption process to comply with the aforementioned rules. The preemption process can utilize the PeRF (400) preemption mechanism including PAUSE / RESUME operations and transmission interrupts. The task maintenance scheduler (424) can include at least one of a Large Message Scheduling Engine (LMSE) (424-2), a Multi-QP Scheduling Engine (Multi-QP Scheduling Engine (MQSE) (424-3), and an Early Completion Engine (ECE) (424-1).

[0079] The Large Message Scheduling Engine (LMSE) (424-2) can handle message-level isolation for small message transmissions. Specifically, the Large Message Scheduling Engine (LMSE) (424-2) can interrupt large message transmissions to allocate resources for small message transmissions. The PeRF (400) preemption mechanism can continuously interrupt large message transmissions to create opportunities for small message transmissions.

[0080] To achieve message-level isolation between B_App transmitting large messages and other applications (M_App and D_App), LMSE (424-2) can use transmission interrupts. The allocation of RNIC resources to applications using small messages can be determined by the number of 0_WAIT WRs inserted between submessages (SUB_MSG_SIZE). This value can be calculated by dividing SUB_MSG_SIZE by 0_WAIT_UNIT.

[0081] For example, suppose the default values ​​for SUB_MSG_SIZE are 16 KB and 0_WAIT_UNIT are set to 1 KB. With these values, a QP using small messages can have 16 times more transmission opportunities than a QP using large messages. Reducing 0_WAIT_UNIT can insert more 0_WAIT WRs between sub WRs, which can increase the RNIC resource allocation for transmitting small messages. Large messages that do not need to be split (≤ SUB_MSG_SIZE) can be processed in the onloading (410) part instead of being passed to the offloading (420) part. This can prevent throughput degradation due to communication overhead between the main thread and background threads. Although the main thread may need to post several 0_WAIT WRs, the impact on application performance can be minimized because the number of 0_WAIT WRs is small.

[0082] The Multi-QP Scheduling Engine (MQSE) (424-3) can manage multi-QP applications to provide QP-level isolation. The MQSE (424-3) can manage the number of active queue pairs of a multi-QP application using PAUSE and RESUME operations. The MQSE (424-3) can use PAUSE / RESUME operations within the PeRF (400) preemption mechanism to regulate the number of QPs used by Appmulti, thereby promoting performance isolation between applications with different QP counts.

[0083] RNIC is a multi-QP application (App multi ) and single QP application (App single), MQSE (424-3) can initiate isolation process at QP level. MQSE (424-3) can flexibly limit the number of active QPs for each tenant by combining PAUSE and RESUME operations. For example, MQSE (424-3) can limit the number of active QPs for all tenants if necessary. allow Set to 1 to App multi The QPs of the PeRF (400) can be activated one by one in a round-robin manner. Accordingly, the PeRF (400) can be activated in the App multi App's traffic single can be isolated from traffic.

[0084] However, MQSE(424-3) is M_App multi M_App may need to support a maximum message rate. multi Parallel use of multiple QPs may be required to achieve the maximum message rate. Therefore, QNum represents the minimum number of QPs required to fully utilize the maximum message rate of the RNIC. capa Taking this into account, MQSE(424-3) starts with the default value of 1 and assigns QNum for each tenant. allow can be adjusted. The application QNum allow When an application attempts to post a WR through multiple QPs exceeding , the MQSE (424-3) can ensure that each QP is given a fair chance to activate in a round-robin manner. If an application attempts to post a WR to an inactive QP, these WRs can be buffered in the MQSE (424-3) queue instead of being delivered directly to the RNIC until the QP becomes active. When the inactive QP becomes active, the MQSE (424-3) can first post the buffered WRs and then allow the application to directly post the WR to the RNIC. This behavior of the MQSE (424-3) allows the PeRF (400) to send the WR to the B_App multi Wow M_App multican be managed fairly to avoid adversely affecting other applications while using multiple QPs.

[0085] The Early Completion Engine (ECE) (424-1) can poll for additional WRs generated by the preemption mechanism to prevent transmission queue saturation. PeRF (400) can generate additional WRs to enforce performance isolation within the preemption mechanism. However, this can saturate the application's SQ, resulting in performance degradation. The ECE (424-1) can periodically poll these additional WRs to reserve space for application WRs and mitigate potential performance issues.

[0086] In RDMA, a Work Request (WR) can be stored in a Transmit Queue (SQ) until the corresponding Completion Queue Element (CQE) is polled by the application. If the SQ is full, preventing further WR postings, the application may need to periodically poll the CQE to make room for future WRs. However, the PeRF worker operates transparently to the application and can post sub-WRs, 0_WAIT WRs, WAIT WRs, or ENABLE WRs that fill the SQ. The ECE (424-1) can be responsible for polling the CQEs corresponding to these transparent WRs. When the ECE (424-1) attempts to poll a CQE in the application's CQ, the regular CQEs that need to be delivered to the application may also be polled. Therefore, the ECE (424-1) may need to buffer these regular CQEs and deliver them to the application later.

[0087] The PeRF preemption mechanism, including the aforementioned onloading (410) process operation and offloading (420) process operation, can operate in GPUDirect RDMA, which is direct communication between remote GPUs used in general RDMA and distributed training of large-scale models. In distributed training of large-scale models, the training task is classified as B_App, and inference task in inference of large-scale models is classified as D_App. In a situation where traffic corresponding to each task coexists, the traffic can be isolated from each other through the preemption mechanism.

[0088] In distributed learning of large-scale models, training tasks require large amounts of data and computation, making them classified as B_Apps. Training tasks require high bandwidth and processing power, and may require the rapid transfer and processing of large amounts of data.

[0089] In contrast, inference tasks for large-scale models process real-time data and generate predictions, and can be categorized as D_Apps. Low latency and high processing speeds can be crucial for inference tasks. Because they are used in applications requiring real-time responses, inference tasks may require fast and efficient data processing.

[0090] The PeRF preemption mechanism can effectively isolate these tasks with different requirements. When training and inference tasks coexist on the same infrastructure, the preemption mechanism can appropriately distribute the network resources required for each task and prevent interference between them.

[0091] The preemption mechanism separates the traffic between B_App and D_App and allocates network resources according to the characteristics of each task, optimizing performance. Large-scale message transmissions for training tasks are processed separately from small-scale message transmissions, ensuring low-latency performance for inference tasks without compromising training performance.

[0092] Therefore, the PeRF preemption mechanism can maintain optimal performance in an environment where distributed training and inference tasks of large-scale models coexist, and enable each task to efficiently use the resources required.

[0093] GPUDirect RDMA enables direct memory access between GPU memory and a network interface card (NIC). GPUDirect RDMA can improve data transfer speeds by minimizing or eliminating CPU intervention in data transfers between the GPU and the NIC.

[0094] Traditional data transfer processes involve copying data from GPU memory to host (CPU) memory, transferring it from host memory to the NIC, and then transferring it from the NIC to the remote GPU. However, GPUDirect RDMA eliminates this intermediate step, enabling data to be transferred directly from GPU memory to the NIC, and then from the NIC to the remote GPU.

[0095] GPUDirect RDMA can be useful for distributed training of large-scale models and real-time data processing. For example, when multiple GPUs train large data sets in parallel, GPUDirect RDMA can optimize inter-GPU communication, reducing training time and improving model efficiency. Furthermore, in applications that process and analyze large amounts of data in real time, GPUDirect RDMA can accelerate data transfer and improve analytics performance.

[0096] Distributed training of large-scale models can involve distributing massive datasets and complex models across multiple computers or nodes to train them in parallel. Distributed training of large-scale models is frequently used in high-performance computing (HPC) environments or data center networks (DCNs). Each node handles a portion of the model and performs training simultaneously, reducing overall training time and maximizing model performance.

[0097] Traffic isolation can be achieved by separating network resources for each task in a network environment where multiple tasks coexist, thereby preventing interference. Traffic isolation can prevent performance degradation for specific tasks and improve overall network efficiency. For example, large-scale message transmission for a learning task can be separated from small-scale message transmission, and network resources can be allocated according to the characteristics of each task to ensure optimal performance.

[0098] A training task (B_App) is the process of training a large-scale distributed learning model, potentially optimizing model parameters using a large dataset. Training tasks require extensive computation and data transfer, potentially requiring high network bandwidth and GPU resources.

[0099] An inference task (D_App) can be the process of processing new data and making predictions using a trained model. Inference tasks prioritize low latency and high processing speed, and can be used in applications that require real-time responses.

[0100]

[0101] FIG. 5 is a diagram for explaining a preemption mechanism of an RNIC in PeRF according to one embodiment.

[0102] The description referring to Fig. 4 can be equally applied to Fig. 5, and overlapping content can be omitted.

[0103] Referring to Figure 5, we can see an RNIC preemption mechanism that can temporarily suspend or resume RNIC resource utilization for each QP. The RNIC preemption mechanism can operate without the need for network resource estimation and may not impede the packet processing performance of the RNIC. Consequently, PeRF (400) can efficiently isolate message transmissions of multiple tenants, providing improved performance compared to existing RDMA isolation methods.

[0104] The RNIC preemption mechanism can use special WRs (ENABLE and WAIT) provided by managed QPs and user-level RDMA APIs.

[0105] In one embodiment, PeRF (400) can perform an RNIC preemption mechanism operation to temporarily suspend or resume resource utilization of the RNIC during an offloading process operation.

[0106] The operation of the preemption mechanism according to one embodiment may include an ENABLE (510) operation and a WAIT operation (520).

[0107] Referring to ENABLE operation (510), in the RNIC preemption mechanism, when an application creates a QP in managed mode, the WR posted to the QP can be stored in host memory and an ENABLE WR can be posted to another QP. The ENABLE WR includes the ID of the target managed QP, and the target managed QP can store the WR initiated to the QP in host memory without directly transmitting it to the RNIC.

[0108] That is, when an application creates a QP in managed mode, any WRs posted to the QP may be stored in host memory rather than being directly transmitted to the RNIC. To enable the RNIC to retrieve the WR, the application may need to post an ENABLE WR to another regular QP, specifying the ID of the target managed QP in the ENABLE WR.

[0109] Referring to the WAIT operation (520), in the RNIC preemption mechanism, PeRF (400) may post a WAIT WR to the QP to cause the QP to stop processing subsequent WRs until the corresponding WR is completed. The WAIT WR may record the number of CQEs to poll and the number of CQs to poll the CQEs.

[0110] That is, a WAIT WR posted to a QP can instruct the RNIC to stop processing subsequent WRs in the QP until the WR is completed. The WAIT WR must record wait_cqe_num and wait_cq, which can indicate the number of CQEs to poll and the CQ from which to poll those CQEs. The WAIT WR can be completed only when the number of CQEs specified in wait_cqe_num has been polled in wait_cq. After the WAIT WR is completed, the RNIC can resume processing subsequent WRs in the QP.

[0111]

[0112] FIG. 6a and FIG. 6b are diagrams for explaining MQSE operation and LMSE operation according to one embodiment.

[0113] The description with reference to FIGS. 4 and 5 can be equally applied to FIGS. 6a and 6b, and overlapping content can be omitted.

[0114] Referring to FIG. 6A, PeRF (400) can initiate the MQSE operation (610) process by creating an additional managed QP, Preemption Control QP (PCQ), and a corresponding CQ, Preemption Control CQ (PCC). Then, PeRF (400) can populate the PCQ by posting a sufficient number of WRs (0_WR) to the host memory space. Here, the WR can request a 0-byte message containing only a header without a payload. The PCQ can be configured to facilitate loopback communication. App multi To manage individual QPs, PeRF (400) can use the above components in conjunction with ENABLE and WAIT WR. During PAUSE operation, PeRF (400) can manage QNum allow App excluding randomly selected QPs multi A WAIT WR can be posted to a selected QP. The WAIT WR is configured to complete when a CQE is polled by the PCC, which allows the RNIC to send a QNum of non-suspended (activated) QPs. allow Only WRs of QPs can be processed. During a RESUME operation, PeRF (400) can post ENABLE WR to one of the active QPs to enable the RNIC to resume WR processing of the oldest QP among the paused QPs (inactive QPs). In particular, after ENABLE WR, WAIT WR can be posted to increase the number of active QPs to QNum. allow can be guaranteed to match.

[0115] Referring to Fig. 6b, since the RNIC processes packets without considering the packet size, PeRF (400) may need to abort large message transmissions in order to isolate small message transmissions. Therefore, to abort large message transmissions, PeRF (400) may perform the LMSE operation (620) process by implementing a transmission interrupt using 0_WAIT WR. 0_WAIT WR may be created by setting wait_cqe_num to 0. When the RNIC encounters a 0_WAIT WR posted to a QP, it immediately completes it, aborts the message transmission of the QP, and may process WRs posted to other QPs. When intercepting a large message transmission request, PeRF (400) may divide the request into multiple sub-WRs that transmit sub-messages of the same size (SUB_MSG_SIZE). PeRF (400) may insert one or more 0_WAIT WRs between the sub-WRs. LMSE(424-2) operation (620) can prevent B_App from excessively using RNIC resources to transmit large messages and allocate reserved resources to M_App or D_App to transmit small messages.

[0116]

[0117] In one embodiment, PeRF's preemption mechanism can incur overhead for PeRF. To mitigate this, the PeRF master can selectively activate isolation processes based on the global tenant distribution, triggering them only when problematic situations arise. For message-level isolation, PeRF can split large messages into chunks larger than 16KB and insert batches of 0_WAIT WRs instead of posting one 0_WAIT WR per smaller chunk. This approach can reduce the CPU overhead of PeRF's message-level isolation. For QP-level isolation, overhead can also be reduced because only a few WAIT and ENABLE WRs are posted.

[0118] By scalable worker thread management, PeRF can offload portions of the workload to a single CPU core for CPU efficiency. For efficient communication between threads, SHM (shared memory) can be implemented as a lock-free data structure. This data structure can ensure that each element is exclusively updated by a single thread. Furthermore, atomic operations can be leveraged to manage information for multithreaded applications where multiple threads simultaneously post WRs to the QP. To avoid CPU contention between PeRF workers, the PeRF master can schedule worker threads based on condition variables, enabling it to support thousands of tenants and QPs in parallel without compromising performance.

[0119] In Fabric isolation, PeRF focuses on solving tenant-level RNIC resource sharing problems, while flow-level fabric sharing can be achieved using existing congestion control mechanisms.

[0120] While the described embodiments primarily focus on the design and implementation of PeRF for WRITE operations, PeRF can also support SEND / RECV and READ operations. SEND operations are similar to WRITE operations, and RECV operations can be implemented by PeRF's isolation process, which can be driven by the sender. For RECV operations, PeRF can be implemented by adding a mechanism to split large message reception WRs and early poll the corresponding CQEs on PeRF workers. Furthermore, to efficiently manage the bidirectional data transfer of READ operations, PeRF can additionally implement RPC-based READ operations. This approach can be adopted by PeRF to address the unique challenges of READ operations, where a remote RNIC generates messages. While PeRF's isolation process must be performed remotely, maintaining low CPU consumption in RDMA environments can be crucial for remote RDMA applications. Therefore, to address this issue, an RPC mechanism can be used to route READ requests to the remote PeRF offloading portion using SEND / RECV operations. This allows the offloading part to efficiently handle READ data transfers using WRITE operations and initiate the isolation process.

[0121]

[0122] FIG. 7 is a block diagram illustrating an electronic device according to one embodiment.

[0123] One or more blocks and combinations of blocks of FIG. 7 may be implemented by a special-purpose hardware-based computer performing a specific function, or by a combination of special-purpose hardware and computer instructions. The descriptions made with reference to FIGS. 1 through 6B may equally apply to FIG. 7. For example, an electronic device (700) according to one embodiment may include a PeRF (400).

[0124] As shown in FIG. 4, the electronic device (700) may include a memory (710) and a processor (720). The electronic device (700) may further include a communication module, and the communication module may include a transmitter and a receiver.

[0125] An electronic device (700) according to one embodiment may include a memory (710) and a processor (720) connected to the memory (710) via a system bus or other suitable circuitry.

[0126] The electronic device (700) may store program code in memory (710). In one embodiment, the memory (710) may include one or more physical memory devices, such as local memory or one or more bulk storage devices. In this case, the local memory may include random access memory (RAM) or other volatile memory devices commonly used while actually executing the program code. The bulk storage device may be implemented as a hard disk drive (HDD), a solid state drive (SSD), or other non-volatile memory device.

[0127] As the executable program code stored in the memory (710) is executed by the electronic device (700), the processor (720) may perform various operations described in the present disclosure. For example, the memory (710) may store program code for causing the processor (720) to perform one or more operations described in FIGS. 1 to 6B.

[0128] Depending on the specific type of device being implemented, the electronic device (700) may include fewer components than those illustrated or additional components not illustrated in FIG. 7. Additionally, one or more of the components may be incorporated into, or otherwise form part of, another component.

[0129] A processor (720) according to one embodiment is a hardware configuration that performs overall control functions for controlling the operations of an electronic device (700). For example, the processor (720) may control the electronic device (700) overall by executing programs stored in a memory (710) within the electronic device (700). The processor (720) may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a neural processing unit (NPU), or the like, provided within the electronic device (700), but is not limited thereto.

[0130] The processor (720) can execute lightweight tasks in the main thread, perform onloading processes that trigger isolation processes, and perform offloading processes that execute complex tasks in background threads. The processor (720) can use the PeRF master to coordinate one or more PeRF workers and perform an offloading process that executes one or more PeRF workers according to the isolation process.

[0131]

[0132] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0133] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0134] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0135] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0136] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0137] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. Onloading process behavior that runs on the main thread, manages lightweight tasks, and triggers isolated processes; and Offloading process behavior that runs in a background thread and performs complex tasks. Including, The above offloading process operation is The operation of coordinating one or more PeRF workers using a PeRF (Preemption-enabled RDMA Framework) master; and An operation of executing one or more PeRF workers according to the above isolation process. How PeRF works, including:

2. In paragraph 1, The above onloading process behavior is The act of intercepting verb calls in an application; An operation to estimate the average message size through a message size estimator; An operation to estimate the number of QPs through a QP (Queue Pair) number estimator; The action of enforcing network policies specified by the system administrator through the Network Policy Manager; and Actions that trigger the above isolation process dynamically depending on the type of application, through the isolation manager. A PeRF operation method including:

3. In paragraph 2, The above quarantine manager A PeRF operating method for monitoring the distribution and network status of global tenants and selectively transmitting a signal to activate the isolation process to the offloading process based on a detected performance abnormality.

4. In paragraph 1, The above PeRF master is A PeRF operation method that monitors the global state of a tenant and creates and schedules the PeRF worker according to the isolation process.

5. In paragraph 1, The above PeRF worker is Application classifier that determines the application type; A task-maintaining scheduler that manages packet transmission and dynamically performs preemption processes. A method of operating PeRF, including:

6. In paragraph 5, The above application classifier is A method of operating PeRF, classifying the type of application based on the estimated message size and number of QPs.

7. In paragraph 5, The above task maintenance scheduler At least one of the Large Message Scheduling Engine (LMSE), the Multi-QP Scheduling Engine (MQSE), and the Early Completion Engine (ECE). A PeRF operation method including:

8. In paragraph 7, The above large-capacity message scheduling engine A PeRF operation method that interrupts large message transmissions to allocate resources for small message transmissions.

9. In paragraph 7, The above multi-queue pair scheduling engine A PeRF operating method for managing the number of active queue pairs in a multi-queue pair application using PAUSE and RESUME operations.

10. In paragraph 7, The above early completion engine A PeRF operation method that polls additional WRs generated by the preemption mechanism to prevent saturation of the transmission queue.

11. In paragraph 1, The above offloading process operation is Includes the operation of the RNIC (Remote Direct Memory Access Network Interface Card) preemption mechanism to temporarily suspend or resume resource utilization of the RNIC, The above preemption mechanism operation is ENABLE and WAIT actions A PeRF operation method including:

12. In paragraph 11, The above ENABLE operation is When an application creates a QP in managed mode, it stores the WR posted to the QP in the host memory and posts an ENABLE WR to another QP. How PeRF works, including:

13. In paragraph 12, The above ENABLE WR is A PeRF operating method, comprising an ID of a target managed QP, wherein the target managed QP stores the WR posted to the QP in host memory without directly transmitting it to the RNIC.

14. In paragraph 11, The above WAIT operation is An action to post a WAIT WR to the above QP to stop processing subsequent WRs in the above QP until the corresponding WR is completed. A PeRF operation method including:

15. In paragraph 14, The above WAIT WR is A method of operating PeRF, which records the number of CQEs to poll and the number of CQs to poll the CQEs.

16. In paragraph 1, The PeRF preemption mechanism including the above onloading process operation and the above offloading process operation is a PeRF operation method that operates in GPUDirect RDMA, which is a direct communication between remote GPUs and GPUs used for distributed training of large-scale models.

17. In paragraph 16, In the distributed learning of the above large-scale model, the learning task is classified as B_App. In the inference of the above large-scale model, the inference task is classified as D_App, A PeRF operation method that isolates traffic from each other through the preemption mechanism in a situation where traffic corresponding to each task coexists.

18. A computer program stored in a computer-readable recording medium to execute the method of claim 1 in combination with hardware.

19. Memory for storing instructions; and One or more processors Including, The above instructions, when executed by the one or more processors, cause the electronic device to: An onloading process that runs on the main thread, manages lightweight tasks, and triggers isolated processes. Perform offloading processes that run in background threads and perform complex tasks, The above offloading process is An electronic device, comprising a process for coordinating one or more PeRF workers using a PeRF master and executing the one or more PeRF workers according to the isolation process.

Images