A system for realizing active-standby mode with multiple RDMA network cards in a securities core trading system

Abstract

The present application relates to the field of RDMA communication technology, and provides a system for realizing master-slave mode with multiple RDMA network cards in a securities core trading system, the system includes a resource management subsystem, the resource management subsystem includes a provider module of user-state virtual devices, the provider module of user-state virtual devices is used to create a virtual bond module, and the virtual bond module is used to manage two real RDMA devices in a bound state; a master-slave switching subsystem, in which the virtual bond module will monitor the bond event of the Linux kernel bond module in the kernel state; an IO transmission subsystem, in which the virtual bond module is used to synchronously copy the sent send / receive queue data to a temporary queue in order to restore the IO data during the subsequent master-slave switching. The present application can deal with the problem of RDMA data transmission failure caused by PCIe device or port failure, has no dependence on upper-layer applications and hardware, and can be adapted to a variety of software tools or hardware.

Description

Technical Field

[0001] The present application relates to the field of RDMA communication technology, and in particular to a system for implementing a master-slave mode with multiple RDMA network cards in a securities core trading system. Background Art

[0002] An RDMA network card is an Ethernet card that integrates RDMA (Remote Direct Memory Access) technology. RDMA technology allows computers in a network to exchange data directly from memory without the intervention of the CPU, thereby significantly improving the efficiency of data transmission, reducing latency, and alleviating the burden on the CPU. In the field of computer networks, "network interface bonding" can be understood as a technical means of bonding multiple network interfaces together to improve network reliability and performance.

[0003] The current mainstream RDMA communication architecture in bond master-slave mode is to establish a bond binding relationship between two ports (physical interfaces) of the same PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) device (Peripheral Component Interconnect Express bus device) to perform RDMA communication with other devices. Compared with the traditional RDMA architecture without bond master-slave mode, this architecture has a certain stability when dealing with emergencies such as port disconnection.

[0004] The existing RDMA communication in bond master-slave mode cannot cross PCIe devices, which means that when a PCIe device has only one port, this solution cannot cover it. In actual production and life, there will be a PCIe device with only one port, or there will be a situation where only one port is available due to a failure of other ports, which cannot be solved by existing technology. At the same time, because both ports are on one card under this solution, when the board fails, both ports may not work, and the risk resistance is weak. At the same time, this solution relies on hardware driver implementation. Currently, the only RDMA device that supports this function is the ConnectX5 board, which is difficult to implement on boards from other manufacturers. Summary of the invention

[0005] In order to help solve the above technical problems, the present application provides a system for implementing a master-slave mode with multiple RDMA network cards in a securities core trading system, which adopts the following technical solutions: A system for implementing a master-slave mode with multiple RDMA network cards in a securities core trading system, wherein the system comprises: Resource management subsystem: The resource management subsystem includes a provider module for user-state virtual devices. The provider module for user-state virtual devices is used to create a virtual bond module. The virtual bond module is used to manage two real RDMA devices in a bound state. When the upper-layer APP creates or destroys RDMA resources, the virtual bond module will simultaneously notify the two real RDMA devices to create or destroy the corresponding RDMA resources. Active / standby switching subsystem: In the active / standby switching subsystem, the virtual bond module in kernel state will monitor the bond events of the Linux kernel bond module. When a switch occurs, the user state will be notified through shared memory. After receiving the switch event, the user state will disconnect the queue of the original master device, and then reconnect the queue of the new master device and resend the data message. IO transmission subsystem,In the IO transmission subsystem, the virtual bond module is used to synchronously copy the sent send / receive queue data to a temporary queue in order to restore the IO data during the subsequent active / standby switch.

[0006] Preferably, the resource management subsystem includes a provider module of user-state virtual devices and a kernel-state virtual device driver interface module that are connected to each other. The provider module of the user-state virtual device is used to create a virtual bond module. When the upper-layer application initializes resources, modifies, or destroys RDMA resources, the virtual bond module will notify the real RDMA device to synchronously create, modify, or destroy the corresponding RDMA resources, and notify the kernel-state virtual device driver module to create RDMA resources for the virtual bond module.

[0007] Preferably, the virtual bond module creates RDMA resources in the following manner: Step S11: The upper layer software calls the interface for creating RDMA resources, and if successful, executes step S21; Step S21: The active side PCle device creates RDMA resources. If successful, execute step S31; Step S31: The backup side PCle device creates RDMA resources. If successful, execute step S41; Step S41: The RDMA software creates RDMA resources. If successful, execute step S51; Step S51: Establish a mapping relationship of RDMA resources according to the PCle device index. If successful, the creation is successful.

[0008] Preferably, the master-standby switching subsystem includes a monitoring module and a master-standby switching management module. The monitoring module is used to register a callback with the Linux kernel bond module and respond to the master-standby switching event in real time. The master-standby switching management module is used to receive the master-standby switching event from the Linux kernel during IO, and disconnect the queue of the backup device and establish the queue of the master device.

[0009] Preferably, the active / standby switching management module receives active / standby switching events from the Linux kernel during IO in the following manner, and performs queue peer disconnection on the standby device and queue peer connection establishment on the active device: Step S21: determine the change status flag, if the change status flag is 1, update the index of the current master device, otherwise execute step S26; Step S22: disconnecting the queue pair connection between the currently non-working device and the peer end; Step S23: Notify the peer device of the updated queue number information of the local end through the TCP transmission control protocol connection; Step S24: Determine whether the local end is performing a sending operation. If so, establish a queue pair connection with the other end and set the state to request to send. If not, establish a queue pair connection with the other end and set the state to ready to receive and execute step S26. Step S25: determine whether there is an unreliable datagram message, if so, execute, if not, resend the message; Step S26: End.

[0010] Preferably, in the IO transmission subsystem, the RDMA software will create a new queue when performing a sending operation. The new queue is used to save the sending queue information and transmission status of the ongoing RDMA transmission. When the master-slave switch occurs, the target device will obtain the previously transmitted sending queue information from the new queue and re-process the sending operation of the work request in the pending transmission state.

[0011] Preferably, the system further comprises an application module, and the application module is used to interact with the underlying virtual bond module through the Verbs unified interface to initiate RDMA operations.

[0012] Preferably, the system further comprises a Linux bond kernel module, which is used to manage the bond status of the physical network card, trigger the active-standby switching process when the active network card fails, and update the active-standby switching status.

[0013] In summary, this application has the following beneficial effects: 1. The present invention proposes a communication solution based on RDMA across PCIe devices in bond master-slave mode, which can better deal with the problem of RDMA data transmission failure caused by PCIe device or port failure.

[0014] 2. The present invention has no dependence on upper layer applications and hardware, and can be adapted to a variety of software tools or hardware; 3. In the case of normal IO or IO when switching occurs, no complex processing logic is involved and the impact on performance is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the RDMA communication architecture in the mainstream bond master-slave mode in the background technology; Figure 2 A schematic block diagram of an embodiment of a system for implementing a master-slave mode with multiple RDMA network cards in a securities core trading system of the present application; Figure 3 A logical diagram of virtual bond resource mapping for this application; Figure 4 A flowchart of a method for creating RDMA resources for a virtual bond module of the present application; Figure 5 This is a post send transmission flow chart in the IO transmission subsystem of this application; Figure 6 This is a schematic diagram of the working process of the master-slave switching module of this application. DETAILED DESCRIPTION

[0016] The present application is further described below in conjunction with the accompanying drawings, and the structure and principle of the present application are very clear to people in the field. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] Here is an explanation of the English in this article: RDMA: Remote Direct Memory Access is a computer network technology that allows two nodes on the network to directly access each other's memory and complete data communication without the involvement of the operating system. It has the advantages of low latency, high bandwidth, and reduced system resource consumption.

[0018] Linux bond: The Linux bond driver provides a way to bundle multiple network interface devices into a single network interface setting for use in network load balancing and network redundancy.

[0019] mr: Memory Region, mr is a memory region, a memory region is a continuous memory managed by RDMA. RDMA controls the access rights of the memory to facilitate remote DMA.

[0020] The provider module is the provider module; Modify is modification; the vbond module is the virtual bond module; active / backup (primary / backup), post send (send operation), dest device (destination device), wr (work request), qp disconnection is queue peer disconnection, qp establish link is queue peer establish link, qpn information is queue number information, post send message is send message, qp connection is queue pair connection.

[0021] RDMA allows user programs to bypass the operating system kernel protocol stack and directly interact with the network card for network communication, thereby achieving high bandwidth and extremely low latency. In addition, RDMA also provides one-sided primitives, which refers to a mechanism or function that allows a node (such as a network card) to perform data transmission or memory access operations without the explicit help or participation of remote nodes. That is, the network card can directly initiate and complete read and write requests to remote memory without the help of remote nodes, which can greatly improve CPU utilization. From the perspective of system software design, RDMA can be directly regarded as a faster network. The current implementation of RDMA mainly includes InfiniBand (InfiniBand), RoCE (RDMA over Converged Ethernet, remote direct memory access based on converged Ethernet) and iWARP (Internet Wide Area RDMA Protocol, Internet Wide Area Remote Direct Memory Access Protocol). Among them, RoCE can achieve high-performance data transmission on Ethernet by encapsulating RDMA operations in Ethernet protocols and is the most widely used. Verbs is a set of standardized programming interfaces that define how applications use RDMA functions.

[0022] To ensure the high availability of the core trading system of securities, in the application scenario, the environment where the core trading system runs will use network card bonding technology. In simple terms, Linux system network card bonding technology virtualizes multiple network cards into one to achieve redundancy; multiple network cards are displayed as one with the same IP address; network configurations will use bonding technology to achieve redundancy at the network port hardware level to prevent single point failures in single network port applications.

[0023] For a bond NIC with multiple physical NICs, one of the physical NICs will be set as the Master, and the others are Slaves. The MAC address of the bond NIC is taken from the physical NIC marked as the Master, and then this MAC address is copied to the other physical NICs.

[0024] In the scenario of the securities core trading system, the network card bond technology uses active backup mode. In this mode, only one slave is activated, and other slave interfaces will be activated only when the interface of the active (working) slave is disconnected. When a failover occurs in the active-backup mode, one or more gratuitous ARPs (gratuitous Address Resolution Protocol) will be sent on the newly activated slave interface. Gratuitous ARPs will be sent on the primary slave interface and all VLAN (Virtual Local Area NetworkInterface) interfaces configured on the interface, and at least one IP address needs to be configured on these interfaces. The gratuitous ARPs sent on the VLAN interface will be attached with the appropriate VLAN id, so this mode provides fault tolerance.

[0025] Figure 1 The figure is a schematic diagram of the mainstream RDMA communication architecture in the bond master-slave mode in the background technology. The current mainstream RDMA communication architecture in the bond master-slave mode is a way to establish a bond relationship between two ports (physical interfaces) of the same PCIe device to perform RDMA communication with other devices. Compared with the traditional RDMA architecture without bond master-slave mode, this architecture has a certain stability when dealing with emergencies such as port down.

[0026] The existing RDMA communication in bond master-slave mode cannot cross PCIe devices, which means that when a PCIe device has only one port, this solution cannot cover it. In actual production and life, there will be a PCIe device with only one port, or there will be a situation where only one port is available due to a failure of other ports, which cannot be solved by existing technology. At the same time, because both ports are on one card under this solution, when the board fails, both ports may not work, and the risk resistance is weak. At the same time, this solution relies on hardware driver implementation. Currently, the only RDMA device that supports this function is the ConnectX5 board, which is difficult to implement on boards from other manufacturers.

[0027] Figure 2This is a schematic block diagram of an embodiment of a system for implementing a network card bond master-slave mode between multiple RDMA network cards in a securities core trading system of the present application. Figure 3 This is a logical diagram of the virtual bond resource mapping for this application. Figure 4 A flowchart of a method for creating RDMA resources for a virtual bond module of the present application is shown below. Figure 5 This is the post send transmission flow chart in the IO transmission subsystem of this application. Figure 6 This is a schematic diagram of the working process of the master-slave switching module of this application.

[0028] Combination Figures 2 to 6 It can be understood that this application adds a virtual RDMA device module (hereinafter referred to as a virtual bond module) in the Verbs module to manage two real RDMA devices in a bound state. When the upper-layer APP creates or destroys resources such as qp (Queue Pair) and cq (Completion Queue), the virtual bond module will simultaneously notify the two real RDMA devices below to create or destroy the corresponding resources.

[0029] In the IO process, the virtual bond module will synchronously copy the sq (Send Queue) / rq (Receive Queue) to be sent.

[0030] Queue data into a temporary queue to prepare for the subsequent active / backup switch (hereinafter referred to as active / backup switch) to restore IO data.

[0031] The virtual bond module will monitor the bond events of the Linux kernel bond module in kernel state. When a switch occurs, it will notify the user state through shared memory. After receiving the switch event, the user state will perform qp disconnection on the original active device, and then perform qp link creation and resend data packets on the new active device.

[0032] The system of this application includes: Resource management subsystem, the resource management subsystem includes a user-mode virtual device provider module and a kernel-mode virtual device driver interface module that are connected to each other. The user-mode virtual device provider module is used to create a virtual bond module. When the upper-layer application initializes resources, modifies (modifies), and destroys RDMA resources, the virtual bond module will notify the real RDMA device to synchronously create, modify, and destroy the corresponding RDMA resources, and notify the kernel-mode virtual device driver module to create RDMA resources for the virtual bond module.

[0033] Specifically, the resource management subsystem is mainly composed of two modules: the provider module of user-mode virtual device verbs and the device driver interface of kernel-mode virtual verbs.

[0034] The main function of the user-mode virtual device provider is to create a virtual bond module to manage two real RDMA devices. When the user-mode APP calls the verbs interface to initialize resources, destroy resources, modify resources, etc., the virtual bond device module will notify the real RDMA device to synchronously create or destroy, modify resources, and notify the kernel-mode virtual device driver module to create resources for the virtual bond device.

[0035] It should be noted that because the RDMA resources of the primary and standby PCIe devices cannot be shared, the virtual bond module maintains a mapping table from virtual RDMA resources to actual RDMA resources. This mapping table uses the virtual bond device resource address to index the resource addresses of the two real RDMA devices and find the corresponding RDMA resources, including qp, cq, mr, pd (ProtectionDomain), ah (Address Handle), etc.

[0036] The active / standby switching subsystem includes a monitoring module and an active / standby switching management module. The monitoring module is used to register callbacks with the Linux kernel bond module and respond to active / backup (active / standby) switching events in real time. The active / backup switching management module is used to receive active / backup switching events from the Linux kernel during IO, and perform qp disconnection on the backup device and qp link establishment on the active device.

[0037] Specifically, this part of the function is one of the cores of this technical solution, which specifically includes a monitoring module for monitoring the bond kernel of the Linux system and a switching management module for managing link breaking, link establishment and resending messages.

[0038] The bond status monitoring module is responsible for registering callbacks with the Linux kernel bond module, responding to active / backup switching events in real time, and sending them to user-mode programs through shared memory.

[0039] The active / standby switching management module is responsible for responding to the switching events received from the kernel during IO, disconnecting the backup device from the qp link, and establishing the qp link with the active device.

[0040] IO transmission subsystem. In the IO transmission subsystem, when the RDMA software performs a post send operation, a new queue will be created. The new queue is used to save the sq (send queue) information and transmission status of the ongoing RDMA transmission. When the master-slave switch occurs, the dest device (destination device) will obtain the previously transmitted sq queue (send queue) information from the new queue, and re-post send the wr (work request) in the waiting state.

[0041] Specifically, when the RDMA software starts post send, it will create a new queue sw sendqueue (abbreviated as sw_sq, software send queue), which will save the sq information of the ongoing RDMA transmission and the transmission status. When the master-slave switch occurs, the dest device will obtain the previously transmitted sq queue information from the sw_sq queue and re-post send the wr that is in the waiting state. Similarly, for the post receive operation, a similar sw receive queue (abbreviated as sw_rq, software receive queue) will be created, and the master-slave switch processing is the same as sw_sq.

[0042] The virtual bond module creates RDMA resources in the following ways: Figure 4 Take qp in RDMA resource as an example): Step S11: The upper layer software calls the interface for creating RDMA resources, and if successful, executes step S21; Step S21: The active side PCle device creates RDMA resources. If successful, execute step S31; Step S31: The backup side PCle device creates RDMA resources. If successful, execute step S41; Step S41: The RDMA software creates RDMA resources. If successful, execute step S51; Step S51: Establish a mapping relationship of RDMA resources according to the PCle device index. If successful, the creation is successful.

[0043] The active / backup switching management module receives active / backup switching events from the Linux kernel during IO in the following ways, and performs qp disconnection on the backup device and qp link creation on the active device: Step S21: determine the change status flag, if it is 1, update the index of the current active device, otherwise execute step S26; Step S22: disconnect the qp connection between the current inactive device (non-working device) and the peer end; Step S23: Notify the peer device of the updated qpn information (Queue Pair Number Information) of the local end through a TCP (Transmission Control Protocol) connection; Step S24: Determine whether the local end is doing post send. If so, establish a rexmit connection with the other end (modify to rts, modify to rts means modify to ready to send state). If not, establish a rexmit connection with the other end (modify to rtr, modify to rtr means modify to ready to receive state) and execute step S26. Rexmit means re-upload data connection. Step S25: Determine whether there is a UD (Unreliable Datagram) message, if so, execute, if not, resend the post send message; Step S26: End.

[0044] In the system of the present application, the user-mode module includes an APP (application module), a software RDMA bond device (virtual bond), and a master-slave switching management module. The APP (application module) interacts with the underlying software RDMAbond device (virtual bond) through the Verbs unified interface to initiate RDMA operations (such as creating qp / cq, sending / receiving data, etc.). The software RDMAbond device (virtual bond) is responsible for managing the resources and IO processes of two real RDMA devices (for dev1 and for dev2, the first real RDMA device and the second real RDMA device), and can maintain a mapping table of virtual resources (qp, cq, mr, etc.) to real device resources. When the APP calls the Verbs interface, the virtual bond module synchronously creates or destroys resources on the two real devices. At the same time, when data is sent (post send) or received (post receive), the data is stored in a temporary queue (sw_sq / sw_rq) for recovering unfinished operations after the master-slave switching. The active / standby switching management module is used to receive switching event notifications from the kernel state (through shared memory), trigger the qp disconnection of the old active device, the qp establishment of the new active device, and resend unfinished data packets from the temporary queue.

[0045] In the system of the present application, the kernel state module includes a virtual device RDMA driver module, a bonding state monitoring module and a Linux bond kernel module. The virtual device RDMA driver module is used to provide an interactive interface with the user state virtual bond module and is responsible for the resource allocation and management of the virtual device. The bonding state monitoring module is used to register a callback with the bond module of the Linux kernel, monitor the master-slave switching events (such as port failures) in real time, and after detecting the switching event, pass the event notification to the user state master-slave switching management module through shared memory. The Linux bond kernel module is used to manage the bond state (active / backup mode) of the physical network card. When the primary network card fails, the switching process is triggered and the active / backup state is updated.

[0046] This application has cross-module communication mode: 1. User-mode and kernel-mode interaction Event notification: The bonding state monitoring module in kernel state sends the master-slave switching event to the master-slave switching management module in user state through shared memory.

[0047] Resource synchronization: The user-state virtual bond module interacts with the kernel-state virtual device RDMA driver through the virtual Verbs device driver interface to achieve synchronous creation and destruction of resources.

[0048] 2. Real device management The virtual bond module directly operates two physical RDMA devices through the real device Verbs provider modules (for dev1 and for dev2).

[0049] In the IO process, the virtual bond module writes data to the queues of two real devices at the same time and restores the data from the temporary queue when the active / standby switch occurs.

[0050] 3. Active / standby switching process Switching trigger: The Linux bond kernel module detects a failure in the primary port → notifies the bonding status monitoring module → notifies the user-mode primary / standby switching management module through shared memory.

[0051] Switching execution: The user-mode module disconnects the qp connection of the old active device → reestablishes the qp connection on the new active device → resends the unfinished data packets from the temporary queue (sw_sq / sw_rq).

Claims

1. A system for implementing a master-slave mode with multiple RDMA network cards in a securities core trading system, characterized in that: The system comprises: Resource management subsystem: The resource management subsystem includes a provider module for user-state virtual devices. The provider module for user-state virtual devices is used to create a virtual bond module. The virtual bond module is used to manage two real RDMA devices in a bound state. When the upper-layer APP creates or destroys RDMA resources, the virtual bond module will simultaneously notify the two real RDMA devices to create or destroy the corresponding RDMA resources. Active / standby switching subsystem: In the active / standby switching subsystem, the virtual bond module in kernel state will monitor the bond events of the Linux kernel bond module. When a switch occurs, the user state will be notified through shared memory. After receiving the switch event, the user state will disconnect the queue of the original master device, and then reconnect the queue of the new master device and resend the data message. IO transmission subsystem,In the IO transmission subsystem, the virtual bond module is used to synchronously copy the sent send / receive queue data to a temporary queue in order to restore the IO data during the subsequent active / standby switch.

2. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 1, characterized in that: The resource management subsystem includes a provider module of user-state virtual devices and a kernel-state virtual device driver interface module that are connected to each other. The provider module of user-state virtual devices is used to create a virtual bond module. When the upper-layer application initializes resources, modifies, or destroys RDMA resources, the virtual bond module will notify the real RDMA device to synchronously create, modify, or destroy the corresponding RDMA resources, and notify the kernel-state virtual device driver module to create RDMA resources for the virtual bond module.

3. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 2, characterized in that: The virtual bond module creates RDMA resources in the following ways: Step S11: The upper layer software calls the interface for creating RDMA resources, and if successful, executes step S21; Step S21: The active side PCle device creates RDMA resources. If successful, execute step S31; Step S31: The backup side PCle device creates RDMA resources. If successful, execute step S41; Step S41: The RDMA software creates RDMA resources. If successful, execute step S51; Step S51: Establish a mapping relationship of RDMA resources according to the PCle device index. If successful, the creation is successful.

4. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 1, characterized in that: The active-standby switching subsystem includes a monitoring module and an active-standby switching management module. The monitoring module is used to register callbacks with the Linux kernel bond module and respond to active-standby switching events in real time. The active-standby switching management module is used to receive active-standby switching events from the Linux kernel during IO, disconnect the queue of the standby device, and establish the queue of the active device.

5. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 4, characterized in that: The active / standby switching management module receives active / standby switching events from the Linux kernel during IO in the following ways, and disconnects the queue peer of the standby device and reconnects the queue peer of the active device: Step S21: determine the change status flag, if the change status flag is 1, update the index of the current master device, otherwise execute step S26; Step S22: disconnecting the queue pair connection between the currently non-working device and the peer end; Step S23: Notify the peer device of the updated queue number information of the local end through the TCP transmission control protocol connection; Step S24: Determine whether the local end is performing a sending operation. If so, establish a queue pair connection with the other end and set the state to request to send. If not, establish a queue pair connection with the other end and set the state to ready to receive and execute step S26. Step S25: determine whether there is an unreliable datagram message, if so, execute, if not, resend the message; Step S26: End.

6. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 1, characterized in that: In the IO transmission subsystem, the RDMA software will create a new queue when performing a sending operation. The new queue is used to save the sending queue information and transmission status of the ongoing RDMA transmission. When the master-slave switch occurs, the target device will obtain the previously transmitted sending queue information from the new queue and re-process the work request in the pending transmission state for the sending operation.

7. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 1, characterized in that: The system also includes an application module, which is used to interact with the lower-layer virtual bond module through the Verbs unified interface to initiate an RDMA operation.

8. The system for realizing active-standby mode with multiple RDMA network cards in the securities core trading system according to claim 1, characterized in that: The system also includes a Linux bond kernel module, which is used to manage the bond status of the physical network card. When the active network card fails, the active-standby switching process is triggered and the active-standby switching status is updated.

Images