A method and device for RDMA dual network card active-standby switching

Abstract

The application discloses a RDMA dual-network card master and backup switching method and device, and comprises the following steps: intercepting the RDMA call of an application program through a self-provided RDMA function library of an intermediate library; if the RDMA call is intercepted, responding to the RDMA call through the self-provided RDMA function library to perform RDMA communication; mapping the resource required by the RDMA communication into the resource of the intermediate library to enable the application program to directly access the resource of the intermediate library; monitoring the change of the RDMA network card through the intermediate library, and performing secondary reconnection through the backup RDMA network card when the master RDMA network card is found to have a fault; and switching the resource of the intermediate library of the associated master RDMA network card into the associated backup RDMA network card to continue the RDMA communication after the reconnection is successful. The application aims to solve the problem that the communication cannot be taken over when the RDMA technology fails, and realizes the fast and stable master and backup switching of the RDMA dual-network card.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer network communication, in particular to a RDMA dual-network card master-slave switching method and device. BACKGROUND

[0002] RDMA (Remote Direct Memory Access) is a hardware I / O technology, which realizes low latency, high throughput and close to zero CPU overhead, and is the benchmark of the current communication performance between data center hosts. Although RDMA can meet the demand of data center application for communication performance, it lacks high availability. The traditional network protocol stack mainly relies on the host implementation, and the connection state and other information are saved in the host memory. Therefore, when a physical network card fails, the host can easily use other physical network cards to continue to maintain communication. The RDMA technology is different, the protocol stack of RDMA is implemented in the RDMA network card, and all communication states are saved in the network card memory. This means that once the RDMA network card fails, all communication states related to it will be invalid, and the communication cannot be directly taken over by other RDMA network cards, which is the main obstacle to realizing the high availability of RDMA. SUMMARY

[0003] The technical problem solved by the present application: In view of the above problems of the prior art, the present application provides a RDMA dual-network card master-slave switching method and device, which aims to solve the problem that the communication of RDMA technology cannot be taken over when the network card fails, and realizes fast and stable master-slave switching for RDMA dual-network card.

[0004] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0005] A RDMA dual-network card master-slave switching method, comprising the following steps: intercepting the RDMA call of an application program through the self-provided RDMA function library of an intermediate library, if the RDMA call is intercepted, responding to the RDMA call through the self-provided RDMA function library to perform RDMA communication, mapping the resources required for RDMA communication as the resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, monitoring the changes of the RDMA network card through the intermediate library, and when it is monitored that the master RDMA network card fails, performing secondary reconnection through the standby RDMA network card, and after the reconnection is successful, switching the resources of the intermediate library associated with the master RDMA network card to the associated standby RDMA network card to continue RDMA communication.

[0006] Optionally, the self-provided RDMA function library of the intermediate library is loaded into the operating system through the LD_PRELOAD environment variable, and after being loaded into the operating system through the LD_PRELOAD environment variable, the self-provided RDMA function library further includes reading a configuration file at a specified path, and parsing a value of a master-slave switching mode configuration item bond_mode in the configuration file; and only when the value of the master-slave switching mode configuration item bond_mode is master-slave switching mode opening, the intermediate library executes monitoring changes of the RDMA network card, and when a failure of the master RDMA network card is monitored, the intermediate library performs secondary reconnection using the standby RDMA network card, and after the reconnection is successful, resources of the intermediate library associated with the master RDMA network card are switched to the standby RDMA network card to continue RDMA communication.

[0007] Optionally, when the resources required for RDMA communication are mapped as resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, the RDMA network card resources are mapped as the resources of the intermediate library: the handle and the unique identifier of all RDMA network cards in the operating system are obtained by calling the verbs interface, and all RDMA device pointers returned by the verbs interface are recorded in an RDMA device mapping table; the RDMA device is initialized for each RDMA network card in the intermediate library, and the memory of the RDMA device structure applied for each RDMA network card when the RDMA device is initialized is recorded in the RDMA device mapping table, and the memory is copied from the memory pointed to by the RDMA device pointer; two specified RDMA network cards are read from the RDMA device mapping table to be used as the master RDMA network card and the standby RDMA network card respectively, and the intermediate library defaults to map the master RDMA network card to the application program, so that the application program defaults to perform RDMA communication through the master RDMA network card.

[0008] Optionally, when the resources required for RDMA communication are mapped as resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, the RDMA connection resources are mapped as the resources of the intermediate library: a dynamic memory HA_cmid is maintained, and when the handle and the unique identifier of all RDMA network cards in the operating system are obtained by calling the verbs interface, the user-mode memory applied for by the verbs interface is copied to the dynamic memory HA_cmid, so that the application program directly accesses the dynamic memory HA_cmid to obtain the RDMA connection identifier pointer cmid, and the dynamic memory HA_cmid is converted into the corresponding RDMA connection identifier pointer cmid in the dynamic library, the handle handle and the kernel-created RDMA connection identifier pointer cmid corresponding to the RDMA connection identifier pointer cmid are matched, and the converted RDMA connection identifier pointer cmid is associated to the RDMA network card accessed by the verbs interface.

[0009] Optionally, when the resources required for the RDMA communication are mapped as resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, the mapping of the memory resources of the RDMA communication as the resources of the intermediate library comprises: first applying corresponding dynamic memories for the memory resources required for the application program RDMA communication, the memory resources comprising a protection domain PD, a completion queue CQ, a queue pair QP, an address handle AH, and an RDMA registration MR, the application of the corresponding dynamic memories comprising a protection domain dynamic memory HA_PD, a completion queue dynamic memory HA_CQ, a queue pair dynamic memory HA_QP, an address handle dynamic memory HA_AH, and an RDMA registration dynamic memory HA_MRM, providing the protection domain dynamic memory HA_PD, the completion queue dynamic memory HA_CQ, the queue pair dynamic memory HA_QP, the address handle dynamic memory HA_AH, and the RDMA registration dynamic memory HA_MRM to the application program for direct access, converting the protection domain dynamic memory HA_PD, the completion queue dynamic memory HA_CQ, the queue pair dynamic memory HA_QP, the address handle dynamic memory HA_AH, and the RDMA registration dynamic memory HA_MRM into corresponding memory resources, and issuing the converted memory resources to the driver, and associating the converted memory resources to the RDMA network card accessed by calling the verbs interface.

[0010] Optionally, the method further comprises: when the secondary reconnection is not successful, re-attempting to call the primary RDMA network card to establish a communication connection to recover the RDMA communication.

[0011] Optionally, the method further comprises: when the secondary reconnection is not successful, re-attempting to call the primary RDMA network card to establish a communication connection to recover the RDMA communication.

[0012] In addition, the application also provides an RDMA dual-network card primary-backup switching device comprising a microprocessor, a memory and two RDMA network cards connected to each other, wherein the microprocessor is programmed or configured to execute the RDMA dual-network card primary-backup switching method.

[0013] In addition, the application further provides a computer readable storage medium, wherein a computer program or instructions are stored, and the computer program or instructions are programmed or configured to execute the RDMA dual network card master-slave switching method by a processor.

[0014] In addition, the application further provides a computer program product, comprising a computer program or instructions, which are programmed or configured to execute the RDMA dual network card master-slave switching method by a processor.

[0015] Compared with the prior art, the application mainly has the following beneficial effects: in view of the problem that the RDMA technology cannot take over communication when a network card fails, the RDMA dual network card master-slave switching method adopts a secondary reconnection mechanism, calls a specific intermediate library through an application program, hijacks functions related to RDMA communication, maps resources required for RDMA communication as resources of the intermediate library, so that the application program directly accesses the resources of the intermediate library, and thus the resources of the intermediate library associated with the master RDMA network card can be switched to the associated backup RDMA network card after successful reconnection to continue RDMA communication, thereby realizing fast and stable master-slave switching. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a basic flowchart of the method of the embodiment of the application.

[0017] Figure 2 It is an example of a modular architecture implementation of the method of the embodiment of the application.

[0018] Figure 3 It is a flowchart of mode configuration in the embodiment of the application.

[0019] Figure 4 It is a resource mapping principle diagram of the intermediate library in the embodiment of the application.

[0020] Figure 5 It is a working flowchart of the reconnection module in the embodiment of the application.

[0021] Figure 6 It is an RDMA communication flowchart between the local device as a sending end and a peer device in the embodiment of the application.

[0022] Figure 7 It is an RDMA communication flowchart between the local device as a receiving end and a peer device in the embodiment of the application. DETAILED DESCRIPTION

[0023] In order to enable personnel in the technical field to better understand the technical solutions of the application, the technical solutions of the application will be further described in detail below with reference to the drawings in the embodiments of the application.

[0024] As Figure 1 shown in the figure, the RDMA dual NIC active-standby switching method of the embodiment includes the following steps: intercepting the RDMA call of an application program through the self-provided RDMA function library of an intermediate library, if the RDMA call is intercepted, responding to the RDMA call through the self-provided RDMA function library to perform RDMA communication, mapping the resources required for the RDMA communication as the resources of the intermediate library so that the application program directly accesses the resources of the intermediate library, monitoring the change of the RDMA NIC through the intermediate library, and when it is monitored that the primary RDMA NIC fails, performing secondary reconnection by using the standby RDMA NIC, and after the reconnection succeeds, switching the resources of the intermediate library associated with the primary RDMA NIC to the resources of the intermediate library associated with the standby RDMA NIC to continue the RDMA communication.

[0025] As Figure 2As shown, the intermediate library in this embodiment includes a self-provided RDMA function library (librdmaha.so), an intermediate library module, a configuration parsing module, a reconnection module, and a fault detection module. The self-provided RDMA function library (librdmaha.so) contains function implementations related to RDMA communication. By modifying the loading path or dynamic link library path of the application program, the application program loads the self-provided RDMA function library (librdmaha.so) at runtime, thereby hijacking the original RDMA function. When the application program calls the RDMA function, it actually calls the function in the self-provided RDMA function library (librdmaha.so). The intermediate library module is the core functional module of the intermediate library, used to map the resources required for RDMA communication to the resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, and to interact with the configuration parsing module, the reconnection module, and the fault detection module. The configuration parsing module is mainly used to configure two RDMA network cards as the primary RDMA network card and the backup RDMA network card, respectively, to ensure that both network cards support RDMA functions and are correctly connected to the network. The fault detection module is implemented through the kernel high availability module rdmaha.ko, used to implement fault detection logic in the intermediate library and detect RDMA network card faults by registering events with the RDMA driver. It interacts with the user-mode intermediate library module through Netlink. When the primary network card fails, the intermediate library will detect the failure through the fault detection module and trigger the switching mechanism. At this time, the intermediate library will call the RDMA communication function of the backup network card to attempt to establish a new communication connection. During the switching process, if the backup network card successfully establishes a communication connection, the intermediate library will update the communication state information in the application program and continue to perform subsequent RDMA communication operations. If the backup network card also fails or cannot establish a connection, the intermediate library will attempt to reconnect the primary network card (if the failure has been recovered) or take other recovery measures. This secondary reconnection mechanism ensures the continuity and stability of communication. Once the backup network card successfully establishes a communication connection, the application program will continue to perform communication operations, ensuring the continuity and stability of data communication. At the same time, the intermediate library will continuously monitor the communication state and perform primary-backup switching again if necessary.

[0026] As Figure 2 and Figure 3As shown, the intermediate library's self-provided RDMA function library (librdmaha.so) is loaded into the operating system through the LD_PRELOAD environment variable, and after being loaded into the operating system through the LD_PRELOAD environment variable, it further includes reading a configuration file at a specified path, and parsing to obtain the value of the primary / backup switching mode configuration item bond_mode in the configuration file. And only when the value of the primary / backup switching mode configuration item bond_mode (taking the value 1 or 0) is the primary / backup switching mode (bond mode) (taking the value 1) is opened, the intermediate library only executes the monitoring of the change of the RDMA network card through the intermediate library, and when the primary RDMA network card fails, it uses the backup RDMA network card for secondary reconnection, and after the reconnection is successful, the resources of the intermediate library associated with the primary RDMA network card are switched to the associated backup RDMA network card to continue RDMA communication. As an optional implementation, the configuration file at the specified path in this embodiment is the configuration file of / etc / rdma / rdmaha.conf, and the parsed configuration information is stored in a global variable for subsequent module use. In addition to the primary / backup switching mode configuration item bond_mode, the related configuration items of the configuration file in this embodiment also include: port space port_space = "RDMA_PS_UDP": parsed as a string type port space (only supports RDMA_PS_UDP or RDMA_PS_TCP), converted to an integer type and saved in a global variable, and the default value is RDMA_PS_UDP. Backup reconnection port reconnect_port = "12321": parsed as a string type backup reconnection port, configured according to the production environment requirements. Bonded network card IP address bond_ip = "198.10.xx.xx": parsed as a string type bonded network card IP address (this is an example), configured according to the actual configuration. Primary / backup RDMA network card global identifier bond_guid = "e8ebd30300f567ee;e8ebd30300f567ba": parsed as a string type primary / backup RDMA network card global identifier, configured according to the actual network card. Multicast address mc_ip = "224.x.x.x": parsed as a string type multicast address (this is an example, only configured in multicast scenarios), configured according to the actual configuration.

[0027] In the implementation of the intermediate library, especially when the system loads the self-provided RDMA function library (librdmaha.so) through the LD_PRELOAD environment variable, two aspects need to be concerned: one is the hijacking function list, that is, which functions are replaced by the same name functions of the intermediate library module; the other is the intermediate resource management and mapping, that is, how to manage and map the RDMA related resources. When the self-provided RDMA function library (librdmaha.so) is loaded, it will provide a series of RDMA related function implementations, which will replace the RDMA library functions originally called by the application program. Since the RDMA library functions are existing public functions, the key of the intermediate library of the embodiment is to use the self-provided RDMA function library of the intermediate library to intercept the RDMA call of the application program, and to increase the mapping of the resources required for RDMA communication to the resources of the intermediate library to enable the application program to directly access the resources of the intermediate library. Therefore, the basic functions of the RDMA functions in the self-provided RDMA function library are the same as the RDMA library functions originally called by the application program, and the specific implementation details are not described here. In the implementation of the master-slave switching function in the RDMA dual-network card environment, the key of the intermediate library module is to map the intermediate resources, so that the application program directly accesses the resources of the intermediate library, prevents the release of the master network card resources from causing the application program to access the memory when the switching occurs, and ensures the continuity and stability of the RDMA communication when the network card is switched.

[0028] As shown in Figure 4 In the embodiment, when the resources required for RDMA communication are mapped to the resources of the intermediate library to enable the application program to directly access the resources of the intermediate library, it includes mapping the RDMA network card resources to the resources of the intermediate library: calling the verbs interface to obtain the handle and unique identifier of all RDMA network cards in the operating system, and recording all RDMA device pointers returned by the verbs interface through the RDMA device mapping table; initializing the RDMA device for each RDMA network card in the intermediate library, and recording the RDMA device structure memory applied for each RDMA network card when initializing the RDMA device through the RDMA device mapping table, which copies the memory pointed to by the RDMA device pointer; reading two specified RDMA network cards from the RDMA device mapping table to be used as the master RDMA network card and the standby RDMA network card respectively, and the intermediate library defaults to map the master RDMA network card to the application program, so that the application program defaults to perform RDMA communication through the master RDMA network card. In the embodiment, the intermediate library defaults to map the master RDMA network card to the application program, so that the application program defaults to perform RDMA communication through the master RDMA network card. Figure 4For example, the current RDMA environment has four RDMA network cards: mlx5_0, mlx5_1, mlx5_2, and mlx5_3. mlx5_0 and mlx5_1 are designated as the primary and backup RDMA network cards, respectively, to ensure mutual high availability. The intermediate library records the RDMA device pointers obtained through the ibv_get_device_list interface. Simultaneously, the intermediate library dynamically allocates memory locations HA_mlx5_0 and HA_mlx5_1 to copy the RDMA device pointers, mapping them to mlx5_0 and mlx5_1 respectively. The intermediate library only maps the primary and backup RDMA network cards.

[0029] like Figure 4 As shown, in this embodiment, when mapping the resources required for RDMA communication to the resources of the intermediate library so that the application can directly access the resources of the intermediate library, the mapping of RDMA connection resources to the resources of the intermediate library includes: maintaining a dynamic memory block HA_cmid; when calling the verbs interface to obtain the handles and unique identifiers of all RDMA network cards in the operating system, copying the memory pointed to by the RDMA connection identifier pointer cmid created synchronously in the kernel by the verbs interface to the dynamic memory HA_cmid, so that the application can directly access the dynamic memory HA_cmid to obtain the RDMA connection identifier pointer cmid; converting the dynamic memory HA_cmid into the corresponding RDMA connection identifier pointer cmid in the dynamic library and sending it to the kernel; and associating the converted RDMA connection identifier pointer cmid with the RDMA network card accessed by calling the verbs interface. RDMA connection mapping associates RDMA connections with specific RDMA network cards. In a dual-NIC environment, this allows the intermediate library module to re-establish or migrate RDMA connections to the new network card when switching network cards. Each RDMA CM connection has a unique identifier—a connection identifier pointer `cmid`, returned by the application's `verbs` interface. This `verbs` interface synchronously creates a new `cmid` in the kernel, which can be mapped to a user-space `cmid` via a `handle`. The intermediate library module maintains a dynamic memory block `HA_cmid`, which copies the memory pointed to by the `cmid`. The application directly accesses the dynamic memory `HA_cmid`, which is translated into the corresponding `cmid` in the intermediate library and then sent to the kernel. The intermediate library module records the network interface card (NIC) information currently used by each RDMA connection, and this information is appended to the corresponding `cmid`. When switching NICs, the intermediate library module migrates the RDMA connection to the new NIC, updating the `cmid` information, including resolving the address, binding to the new NIC, exchanging connection information, and establishing the connection.

[0030] RDMA communication will use rdma resources: PD (protection domain), CQ (completion queue), QP (queue pair), AH (address handle), MR (RDMA registered memory), these memory resources are associated with a specific RDMA network card, when the master and backup switching occurs, continuing to access these memory resources will generate access errors. The intermediate library maps these resources to the intermediate library module, which does not ensure the correctness of memory access. For example Figure 4As shown, in the embodiment, when the resources required for RDMA communication are mapped as the resources of the intermediate library so that the application program directly accesses the resources of the intermediate library, the memory resources for RDMA communication are mapped as the resources of the intermediate library: first, the corresponding dynamic memory is applied for the memory resources required for the application program RDMA communication, the memory resources include a protection domain PD, a completion queue CQ, a queue pair QP, an address handle AH and an RDMA registration MR, the corresponding dynamic memory includes a protection domain dynamic memory HA_PD, a completion queue dynamic memory HA_CQ, a queue pair dynamic memory HA_QP, an address handle dynamic memory HA_AH and an RDMA registration dynamic memory HA_MRM, the protection domain dynamic memory HA_PD, the completion queue dynamic memory HA_CQ, the queue pair dynamic memory HA_QP, the address handle dynamic memory HA_AH and the RDMA registration dynamic memory HA_MRM are provided to the application program for direct access, and the protection domain dynamic memory HA_PD, the completion queue dynamic memory HA_CQ, the queue pair dynamic memory HA_QP, the address handle dynamic memory HA_AH and the RDMA registration dynamic memory HA_MRM are converted into the corresponding memory resources and issued to the driver, and the converted memory resources are associated with the RDMA network card accessed by the calling verbs interface. The application program needs to register the related resources: PD, CQ, QP, MR, etc. before communication: the memory resource pointers can be obtained through the verbs interface such as ibv_alloc_pd, ibv_create_cq, ibv_create_qp and ibv_reg_mr, the intermediate library module applies the corresponding dynamic memory (protection domain dynamic memory HA_PD, completion queue dynamic memory HA_CQ, queue pair dynamic memory HA_QP, address handle dynamic memory HA_AH and RDMA registration dynamic memory HA_MRM), etc., and the memory pointed to by the memory resource pointer is copied to the intermediate resource applied by the intermediate library. The application program directly accesses the corresponding dynamic memory of the intermediate library. When switching occurs, the intermediate library module applies the memory resources of the new network card, and copies the resources to the dynamic memory corresponding to the intermediate library, and the application program uses the resources, avoiding resource error access caused by network card switching. In the RDMA communication process, the intermediate library module needs to ensure that the memory access correctly accesses the currently active RDMA network card. The memory accessed by the application program is the internal memory of the intermediate library, which always points to the context of the network card for current communication, and the memory is pointed to the correct network card context, and this operation is completed by the intermediate library when switching occurs. When the RDMA communication ends, the intermediate library module needs to be responsible for releasing the registered memory area, and updating the memory resource mapping table.

[0031] The fault detection module embodiment realizes the monitoring and notification of the RDMA network card state by combining the kernel module and the user state intermediate library, using the verbs interface and the Netlink communication mechanism. This scheme can flexibly handle the network card switching event and ensure the continuity and stability of the RDMA communication. The flow chart of the fault detection module is shown in Figure 5 The fault detection module embodiment realizes the monitoring and notification of the RDMA network card state by combining the kernel module and the user state intermediate library, using the verbs interface and the Netlink communication mechanism. This scheme can flexibly handle the network card switching event and ensure the continuity and stability of the RDMA communication. The flow chart of the fault detection module is shown in Figure 5 The fault detection module registers a client by calling the verbs registration interface, which allows the application to directly interact with the RDMA hardware. By registering a client, the notification of the RDMA hardware event can be obtained: driver registration, deregistration. The fault detection module adopts the Netlink communication method: Netlink allows the kernel to send messages to the user state process and receive responses from the user state process. The tasks of the fault detection module include: master and backup RDMA network card state change notification: when the master and backup RDMA network cards occur UP event (.add = ha_add_one) / DOWN event (.remove = ha_remove_one), the kernel module will capture these events and send notifications to the user state intermediate library through Netlink, and if a DOWN event occurs, it is determined that the RDMA network card has failed. These notifications contain the state information of the network card, such as whether it has been activated, whether it has failed, etc. Update the intermediate library RDMA device mapping table and device state: after receiving the Netlink notification, the user state intermediate library will update the RDMA device mapping table and the device state according to the information in the notification. Decide whether switching needs to occur: after updating the device mapping table and the state, the user state intermediate library will decide whether the network card switching needs to occur. If switching is needed, the intermediate library will execute the reconnection module.

[0032] In this embodiment, the intermediate library monitors the changes of the RDMA network card and uses the backup RDMA network card for secondary reconnection when the master RDMA network card fails. After successful reconnection, the resources of the intermediate library associated with the master RDMA network card are switched to the backup RDMA network card to continue the RDMA communication, including the intermediate library determining whether the local device is the sending end or the receiving end in the RDMA communication. As shown in Figure 6If the local device (reconnection initiator) is a sending end, the RDMA communication between the local device and the peer device (reconnection listener) in RDMA communication includes: ① the intermediate library (reconnection module) of the local device listens to the master-backup switching signal, and suspends program execution in the intermediate library hijacking function; ② the intermediate library (reconnection module) of the local device initiates a reconnection signal to the peer device; ③ the intermediate library (listening thread) of the peer device listens to the reconnection signal, and suspends program execution in the intermediate library hijacking function; ④ the application program of the peer device sends a waiting connection completion signal to the intermediate library to continue receiving data; ⑤ the intermediate library (listening thread) of the peer device exchanges connection information with the intermediate library (reconnection module) of the local device to reestablish the connection; ⑥ the intermediate library (reconnection module) of the local device sends a reconnection completion signal to the hijacking function; ⑦ the application program of the local device continues RDMA communication to send data to the application program of the peer device; as shown in Figure 7 If the local device (reconnection initiator) is a receiving end, the RDMA communication between the local device and the peer device (reconnection listener) in RDMA communication includes: ① the intermediate library (reconnection module) of the local device listens to the master-backup switching signal, and suspends program execution in the intermediate library hijacking function; ② the intermediate library (reconnection module) of the local device initiates a reconnection signal to the peer device; ③ the intermediate library (listening thread) of the peer device listens to the reconnection signal, and suspends program execution in the intermediate library hijacking function; ④ the application program of the peer device sends a waiting connection completion signal to the intermediate library (listening thread) to continue sending data; ⑤ the intermediate library (listening thread) of the peer device exchanges connection information with the intermediate library (reconnection module) of the local device to reestablish the connection; ⑥ the intermediate library (reconnection module) of the local device sends a reconnection completion signal to the hijacking function; ⑦ the application program of the local device continues RDMA communication to notify the peer device to continue sending data; ⑧ the application program of the peer device continues RDMA communication to send data to the application program of the local device. During the reconnection process, the local device (reconnection initiator) and the peer device (reconnection listener) may suspend sending and receiving control messages and jump to the ibv_post_send / recv function to wait, the listening thread monitors the reconnection progress, and after the reconnection is completed, the main thread of the application program starts the sending and receiving of messages. For the data messages being processed, the sender and receiver will suspend the current data sending and receiving function, and start the reconnection module. Once the reconnection is completed, the listening thread will notify the server to continue executing the previously suspended data sending and receiving function. Figure 6 and Figure 7In the embodiment, each RDMA function is an existing known RDMA function, and specific explanations are as follows: rdma_create_id: create an RDMA event channel and a communication identifier rdma_cm_id, which is a basic handle for subsequent operations. rdma_resolve_addr: resolve an IP address of a remote node into a usable RDMA address (GID, LID, etc.). rdma_resolve_route: find an optimal path to a target node in a local subnet according to a resolved address. ibv_alloc_pd: allocate a protection domain (PD), and all local resources (QP, MR, CQ, etc.) must be associated with the same PD. ibv_create_cq: create a completion queue (CQ) for receiving completion events of work requests. ibv_create_qp: create a queue pair (QP), which is an endpoint for RDMA to perform data transmission. ibv_reg_mr: register a memory region (MR), lock a user-mode buffer, and grant local and remote access permissions. rdma_connect: a client calls and establishes a reliable connected RC or unreliable connected UD with a remote end. ibv_post_send: deliver a send work request (WR) to a send queue of the QP, and trigger an RDMA sending / writing / reading operation. rdma_bind_addr: a server calls, binds the rdma_cm_id to a local IP and port, and prepares for listening. rdma_listen: put the bound address into a listening state, and wait for a client connection request. rdma_accept: accept a client connection request, complete a three-way handshake, and establish an RC connection. ibv_poll_cq: actively poll the completion queue, and check whether there is a completed work request (receiving / sending). Destroy: release all resources (QP, CQ, MR, PD, id, etc.) created in the foregoing, and prevent memory leakage.

[0033] As shown in Figure 1 In the embodiment, the intermediate library monitors changes of the RDMA network card, and when it is monitored that the main RDMA network card fails, the standby RDMA network card is used for secondary reconnection. When the reconnection is unsuccessful, the main network card is called again to establish a communication connection to recover the RDMA communication.

[0034] In summary, the RDMA dual NIC active-standby switching method of the embodiment maps the resources required for RDMA communication as resources of the intermediate library so that the application program directly accesses the resources of the intermediate library. The intermediate library resource management and mapping refers to mapping the NIC resources to the intermediate library. After the relevant resources (such as device handles, connection handles, memory resource handles, etc.) are mapped to the intermediate library, the application program can obtain these resources by calling the verbs interface. In the active-standby switching scenario, the intermediate library is responsible for updating the resource mapping and redirecting the resources originally pointing to the primary NIC to the standby NIC. This process is transparent to the application program, that is, the application program is not aware of it and does not need to be additionally configured or modified. In addition, a listening thread and a reconnection module are designed in the secondary connection scheme. The listening thread is a thread running in the background, which is responsible for listening to the state of the network connection. When the opposite end initiates a connection actively, the listening thread can detect the connection request and trigger the reconnection mechanism. The reconnection module is a logic that automatically attempts to reestablish a connection after the primary connection is disconnected. When it detects that reconnection is needed, it will pause the function being executed and start the reconnection module. Once the reconnection is completed, it will continue to execute the function that was previously paused. The RDMA dual NIC active-standby switching method of the embodiment can improve the stability and reliability of the RDMA system in high-load and high-concurrency scenarios. This is specifically manifested in the following aspects: fast switching: the secondary reconnection mechanism and intermediate library hijacking technology are used to realize fast active-standby switching in the RDMA dual NIC environment. strong stability: the intermediate library provides stable fault detection and switching logic to ensure the continuity and stability of communication. good compatibility: the scheme has little impact on the application program and only needs to load the intermediate library to realize the function, which has good compatibility. easy to maintain: the design of the intermediate library separates the fault detection and switching logic from the application program, which is convenient for subsequent maintenance and upgrade.

[0035] In addition, the embodiment also provides an RDMA dual NIC active-standby switching device, which comprises a microprocessor, a memory and two RDMA NICs connected with each other, and the microprocessor is programmed or configured to execute the RDMA dual NIC active-standby switching method. The embodiment also provides a computer readable storage medium, which stores a computer program or instructions programmed or configured to execute the RDMA dual NIC active-standby switching method by a processor. The embodiment also provides a computer program product comprising a computer program or instructions programmed or configured to execute the RDMA dual NIC active-standby switching method by a processor.

[0036] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0037] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for RDMA dual-NIC master / slave handover, characterized in that, The process includes the following steps: intercepting RDMA calls from applications using the built-in RDMA function library of the intermediate library; if an RDMA call is intercepted, responding to the RDMA call using the built-in RDMA function library to perform RDMA communication; mapping the resources required for RDMA communication to the resources of the intermediate library so that applications can directly access the resources of the intermediate library; monitoring changes to the RDMA network card through the intermediate library; and if a failure is detected in the primary RDMA network card, using the backup RDMA network card for a second reconnection; and after a successful reconnection, switching the resources of the intermediate library of the associated primary RDMA network card to the associated backup RDMA network card to continue RDMA communication. The process of mapping the resources required for RDMA communication to the resources of the intermediate library so that the application can directly access the resources of the intermediate library includes mapping the RDMA network card resources to the resources of the intermediate library: calling the verbs interface to obtain the handles and unique identifiers of all RDMA network cards in the operating system, and recording the RDMA device pointers returned by all verbs interfaces through the RDMA device mapping table. In the intermediate library, an RDMA device is initialized for each RDMA network card. The RDMA device mapping table records the memory allocated for the RDMA device structure for each RDMA network card during the initialization of the RDMA device. This memory is copied from the memory pointed to by the RDMA device pointer. The system reads two specified RDMA network cards from the RDMA device mapping table to serve as the primary RDMA network card and the backup RDMA network card, respectively. The intermediate library maps the primary RDMA network card to the application by default, so that the application will use the primary RDMA network card for RDMA communication by default.

2. The RDMA dual-NIC master / slave handover method according to claim 1, characterized in that, The built-in RDMA function library of the intermediate library is loaded into the operating system via the LD_PRELOAD environment variable. After being loaded into the operating system via the LD_PRELOAD environment variable, it also includes reading the configuration file at the specified path, parsing and obtaining the value of the primary / standby switchover mode configuration item bond_mode in the configuration file; and only when the value of the primary / standby switchover mode configuration item bond_mode is enabled, the intermediate library will perform the following: monitor changes of the RDMA network card through the intermediate library, and if a failure is detected in the primary RDMA network card, use the standby RDMA network card to reconnect for a second time, and after a successful reconnection, switch the resources of the intermediate library of the associated primary RDMA network card to the associated standby RDMA network card to continue RDMA communication.

3. The RDMA dual-NIC master / slave handover method according to claim 1, characterized in that, The process of mapping the resources required for RDMA communication to the resources of the intermediate library so that the application can directly access the resources of the intermediate library includes mapping RDMA connection resources to the resources of the intermediate library: maintaining a dynamic memory block HA_cmid; when calling the verbs interface to obtain the handles and unique identifiers of all RDMA network cards in the operating system, copying the user-space memory requested by the verbs interface to the dynamic memory HA_cmid, so that the application can directly access the dynamic memory HA_cmid to obtain the RDMA connection identifier pointer cmid; and converting the dynamic memory HA_cmid into the corresponding RDMA connection identifier pointer cmid in the dynamic library. The handle that comes with the RDMA connection identifier pointer cmid is matched with the RDMA connection identifier pointer cmid created by the kernel, and the converted RDMA connection identifier pointer cmid is associated with the RDMA network card accessed by calling the verbs interface.

4. The RDMA dual-NIC master / slave handover method according to claim 1, characterized in that, The process of mapping the resources required for RDMA communication to the resources of the intermediate library so that the application can directly access the resources of the intermediate library includes mapping the memory resources of RDMA communication to the resources of the intermediate library: First, dynamic memory corresponding to the memory resources required for RDMA communication by the application is requested. The memory resources include protection domain PD, completion queue CQ, queue pair QP, address handle AH, and RDMA registration MR. The corresponding dynamic memory requested includes dynamic memory for protection domain HA_PD, dynamic memory for completion queue HA_CQ, dynamic memory for queue pair HA_QP, dynamic memory for address handle HA_AH, and dynamic memory for RDMA registration MR. The HA_MRM memory provides the application with direct access to the dynamic memory of the protection domain (HA_PD), the dynamic memory of the completion queue (HA_CQ), the dynamic memory of the queue pair (HA_QP), the dynamic memory of the address handle (HA_AH), and the dynamic memory of the RDMA registration (HA_MRM). It also converts the dynamic memory of the protection domain (HA_PD), the dynamic memory of the completion queue (HA_CQ), the dynamic memory of the queue pair (HA_QP), the dynamic memory of the address handle (HA_AH), and the dynamic memory of the RDMA registration (HA_MRM) into corresponding memory resources and sends them to the driver. Finally, it associates the converted memory resources with the RDMA network card accessed by calling the verbs interface.

5. The RDMA dual-NIC master / slave handover method according to claim 1, characterized in that, The process of monitoring changes in the RDMA network card through the intermediate library and, upon detecting a failure of the primary RDMA network card, using the backup RDMA network card for a secondary reconnection, and switching the resources of the intermediate library associated with the primary RDMA network card to the associated backup RDMA network card to continue RDMA communication after a successful reconnection, includes the intermediate library determining whether the local device is the sender or receiver in the RDMA communication: If the local device is the sender, the RDMA communication between the local device and the peer device includes: ① After the intermediate library of the local device detects the primary / backup switch signal, it suspends program execution in the intermediate library hijacking function; ② The intermediate library of the local device initiates a reconnection signal to the peer device; ③ The intermediate library of the peer device listens for the reconnection signal and suspends program execution in the intermediate library hijacking function; ④ The application program of the peer device sends a waiting connection completion signal to the intermediate library to continue receiving data; ⑤ The intermediate library of the peer device exchanges connection information with the intermediate library of the local device to re-establish the connection; ⑥ The intermediate library of the local device sends a signal to the hijacking function. The reconnection complete signal; ⑦ The application of the local device continues RDMA communication to send data to the application of the peer device; If the local device is the receiving end, the RDMA communication between the local device and the peer device in RDMA communication includes: ① After the local device's intermediate library detects the master / slave switchover signal, it suspends program execution in the intermediate library hijacking function; ② The local device's intermediate library initiates a reconnection signal to the peer device; ③ The peer device's intermediate library listens for the reconnection signal and suspends program execution in the intermediate library hijacking function; ④ The peer device's application sends a waiting connection complete signal to the intermediate library to continue sending data; ⑤ The peer device's intermediate library exchanges connection information with the local device's intermediate library to re-establish the connection; ⑥ The local device's intermediate library sends a reconnection complete signal to the hijacking function, enabling the application to continue receiving data; ⑦ The local device's application continues RDMA communication to notify the peer device to continue sending data; ⑧ The peer device's application continues RDMA communication to send data to the local device's application.

6. The RDMA dual-NIC master / slave handover method according to claim 1, characterized in that, The method of monitoring changes in the RDMA network card through the intermediate library and using the backup RDMA network card for secondary reconnection when the primary RDMA network card fails is also included. If the reconnection fails, the method of retrying to call the primary network card to establish a communication connection to restore RDMA communication is also included.

7. An RDMA dual-NIC master / slave switching device, comprising a microprocessor, a memory, and two RDMA NICs interconnected, characterized in that, The microprocessor is programmed or configured to perform the RDMA dual-NIC master / slave switching method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the RDMA dual network card primary / backup switching method according to any one of claims 1 to 6 via a processor.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the RDMA dual network card primary / backup switching method according to any one of claims 1 to 6 via a processor.

Images