Reliable multicast transmission method oriented to RoCE protocol

Abstract

The invention relates to the technical field of communication, in particular to an RoCE protocol-oriented reliable multicast transmission method, which comprises the following steps of: S1, dividing a multicast source and a multicast group in a hierarchical manner; s2, a sending end sends control type information through a UDP protocol, a receiving end monitors a control message, connection is established through an RDMA CM API, and QP ready state conversion is completed; s3, the sending end and the receiving end adopt an RC mode to carry out reliable data transmission, a forwarding node carries out concurrent forwarding in a Reactor multi-thread mode, and a main thread polls and receives a CQ event, determines a forwarding target node, copies message content to a sending buffer area and initiates asynchronous operation; and S4, carrying out API (Application Program Interface) development, and realizing SEND-RECEIVE bilateral operation and READ-WRITE unilateral operation in a multicast scene. The method has the advantages that a reliable multicast transmission function is realized, and a many-to-many concurrent forwarding scene is supported.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a reliable multicast transmission method oriented to a RoCE protocol. Background Art

[0002] In the field of financial information services, the market system has an increasing demand for high-throughput and low-latency technical architecture. To meet this challenge, the market system is introducing more advanced network technologies and facilities to reduce the latency of market information transmission and enhance system performance. In order to better adapt to market demand and improve user satisfaction, many companies have begun to study RDMA technology, which has shown significant advantages in improving data transmission efficiency, reducing latency and enhancing system stability, and has great application prospects.

[0003] At present, the market system mainly transmits data through the traditional TCP / IP network communication mode (BSD Socket). UDP multicast is widely used in the market system due to its efficient data distribution capabilities, ensuring that multiple receivers can receive real-time market information almost simultaneously and achieve data synchronization. However, some existing multicast solutions still face problems such as hardware limitations, packet loss and disorder, which increase the delay of data transmission or processing, making it difficult to ensure that the receiver can receive data with low latency and reliability.

[0004] As an efficient network communication technology, RDMA supports multiple network protocols, including InfiniBand, iWARP, and RoCE (RDMA over Converged Ethernet). The RoCE protocol can run on the existing Ethernet infrastructure, avoiding the overhead of network stack processing, significantly reducing network communication latency and alleviating CPU burden, which is very suitable for efficient data transmission and production environment requirements of the market system. The transmission methods under the RoCE protocol mainly include unreliable datagram transmission (UD) and reliable connection (RC) transmission mode. The UD mode supports multicast and has high transmission efficiency and throughput, but has low reliability and the risk of data packet loss or disorder. It does not support RDMA unilateral READ and WRITE operations and is not suitable for low-latency application scenarios with high security requirements. The RC mode provides higher transmission reliability, similar to TCP connection, and can provide efficient data transmission and low latency characteristics. It is usually used for point-to-point communication.

[0005] In order to meet the market system's demand for high-performance network communication, while balancing transmission efficiency and reliability, low-latency multicast function in many-to-many forwarding scenarios is implemented through RDMA technology, avoiding problems such as packet loss, errors, duplication and disorder, and meeting the localization requirements in actual production deployment. This is an issue worthy of in-depth study. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a reliable multicast transmission method for the RoCE protocol, which is used to realize the reliable multicast transmission function through UDP control class messages and RDMA one-to-one reliable connection (RC) mode, while supporting many-to-many concurrent forwarding scenarios, effectively utilizing the outstanding characteristics of RDMA, and meeting the low-latency reliable multicast scenario application of the market system.

[0007] In order to achieve the above purpose, a reliable multicast transmission method for RoCE protocol is designed, which includes: S1. Multicast group division: According to the actual market forwarding scenario, the multicast source and multicast group are divided in a hierarchical manner, and the nodes are divided into sending nodes, forwarding nodes and receiving nodes. The sending node, as a multicast source, is only responsible for producing and sending data; the forwarding node is responsible for receiving messages from the sending node or other forwarding nodes, and forwarding them to other forwarding nodes or receiving nodes; the receiving node, as the end of the multicast group, only aggregates and receives the received data; S2. Control plane: The sending end sends control information in the user datagram protocol UDP, sends its own Internet protocol IP and port to the receiving end, and the receiving end listens to the control message, synchronizes the application layer routing table, and creates a control plane for building a multicast group. The communication endpoint EP of the established communication connection is established, and the connection is established through the application programming interface API of the communication manager CM in the remote direct memory access technology RDMA, and the ready state conversion of the work queue to QP is completed; S3. Data plane: the sender and the receiver use the reliable connection mode RC mode for reliable data transmission, and the forwarding node uses the reactor multi-threaded mode Reactor multi-threaded mode for concurrent forwarding. The main thread polls the reception completion queue CQ event, determines the forwarding target node according to the application layer routing table, copies the message content to the sending buffer and initiates an asynchronous operation, performs the send-receive bilateral operation SEND-RECEIVE operation and the read-write unilateral operation READ-WRITE operation; S4. Interface call: C and JAVA API interface development is carried out to realize the send-receive bilateral operation SEND-RECEIVE operation and the read-write unilateral operation READ-WRITE operation of C and JAVA programs in multicast scenarios, and the number of data sent or received or read or written, the size of the data packet and the number of times the data is sent are set through parameters.

[0008] Preferably, the method of the present invention also includes other technical features, wherein in the control plane step, the sending end includes a sending node and a forwarding node, the sending end uses the User Datagram Protocol UDP protocol as an envelope to send control information, and sends its own Internet Protocol IP and port to the receiving end, the receiving end includes a receiving node and a forwarding node, the receiving end monitors control messages, synchronizes the application layer routing table, and sequentially creates a communication endpoint EP for establishing a communication connection, establishes a connection processing handshake process based on the communication endpoint EP information and the application programming interface API, creates a work queue pair QP on the communication endpoint EP, associates the work queue pair QP with the communication endpoint through the application programming interface API, and completes the ready state transition of the work queue pair QP.

[0009] Preferably, the method of the present invention also includes other technical features, wherein in the data plane step, the sending end and the receiving end adopt the reliable connection mode RC mode to reliably transmit data, and when the forwarding node forwards data, it performs concurrent forwarding in the reactor multi-threaded mode Reactor multi-threaded mode, initializes an instance responsible for listening to and distributing network events, registers events for each work queue QP, and the main thread polls the reception completion queue CQ event. After receiving the message, the forwarding target node is determined according to the application layer routing table, and a thread is allocated from the thread pool for each forwarding connection. The message content is copied to the sending buffer and an asynchronous operation is initiated. When performing a two-end send-receive bilateral operation SEND-RECEIVE operation, the sending end data is encapsulated into a data packet that can be transmitted on the link, and the receiving end parses the data. When performing a single-end read-write unilateral operation READ-WRITE operation, after obtaining the destination address through the send-receive bilateral operation SEND-RECEIVE operation, the remote memory can be directly read and written without the need for the remote CPU to participate in sensing the data being read and written in the memory.

[0010] Preferably, the method of the present invention also includes other technical features, wherein in the interface calling step, C and JAVA API interface development are respectively performed to implement a set of data transmission solutions, supporting C and JAVA programs sending and receiving send-receive bilateral operations SEND-RECEIVE operations, read-write unilateral operations READ-WRITE operations of unilateral read READ and unilateral write WRITE operations in multicast scenarios, and setting the number of data items sent or received or read or written, data packet size and number of transmission times through parameters.

[0011] Compared with the prior art, the present invention has the following advantages: Compared with the prior art, the advantages of the present invention are embodied in the following aspects: 1) It effectively integrates the UDP transmission protocol in the traditional communication mode and the reliable transmission mode RC in the RDMA technology, which not only meets the demand for multicast functions in the production environment, but also ensures the integrity and sequence of data transmission, realizes low-latency network communication, and ensures the reliability of data transmission.

[0012] 2) Supports multi-threaded concurrent processing in forwarding scenarios, reduces the delay caused by simulating many-to-many connections, improves the system's ability to handle high-concurrency requests, and optimizes resource utilization efficiency.

[0013] 3) It provides C language and JAVA language calling interfaces, allowing different users to use or integrate this technology more conveniently, enhancing the universality and ease of use of the technology.

[0014] 4) Adapt to domestic hardware equipment, develop based on Ethernet, and support RoCE v2 protocol, the implementation solution has certain scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 , is a flow chart of the method; Figure 2 , is a schematic diagram of multicast group division; Figure 3 , is the application layer routing table diagram; Figure 4 , is the sending node workflow diagram; Figure 5 , is a forwarding node workflow diagram; Figure 6 , is the accepting node workflow diagram; Figure 7 , is a timing diagram; Figure 8 , is a schematic diagram of data message flow. DETAILED DESCRIPTION

[0016] In order to make the purpose, principle and structure of the present invention more clear, the following is further described in conjunction with the accompanying drawings and specific embodiments. The following implementation method takes the use of the method of the present invention in a market system in a production environment as an example to illustrate the present invention.

[0017] Figure 1 This application embodiment is based on a domestic server and an operating system, and uses the method of the present invention to achieve reliable multicast.

[0018] It is worth mentioning that: UDP protocol refers to User Datagram Protocol, which is a connectionless, simple transport layer protocol that provides services based on the IP protocol.

[0019] Reactor multi-threaded mode, also known as reactor multi-threaded mode, is an event-driven model in network programming.

[0020] Based on the current situation of the market system, the present invention provides a reliable multicast transmission method for the RoCE protocol, which combines the UDP protocol multicast function and the RDMA RC reliable transmission characteristics at the application layer, supports low-latency concurrent processing of many-to-many forwarding scenarios, provides C and JAVA interface calls, and meets the deployment environment requirements based on domestic hardware equipment. The method is as follows: S1. Division of multicast groups. According to the actual market forwarding scenario, multicast sources and multicast groups are divided in a hierarchical manner, and nodes are divided into sending nodes, forwarding nodes, and receiving nodes. Specifically, the sending node, as a multicast source, is only responsible for producing and sending data; the forwarding node is responsible for receiving messages from the sending node or other forwarding nodes and forwarding them to other forwarding nodes or receiving nodes; the receiving node, as the end of the multicast group, only aggregates and receives the received data.

[0021] S2. Control plane. In the divided area, the sender (sending node / forwarding node) uses the UDP protocol as an envelope to send control information, and sends its own IP and port to the receiving end (receiving node / forwarding node). The receiving end listens to the control message, synchronizes the application layer routing table, and creates the communication endpoint EP for establishing the communication connection in sequence. The connection processing handshake process is established based on the communication endpoint EP information and the RDMA CM API, and a work queue pair QP is created on the EP. The QP is associated with the communication endpoint through the RDMA CM API, and the QP ready state transition is completed.

[0022] S3. Data plane. Based on the established connection, the sender and the receiver use the RC mode for reliable data transmission. When the forwarding node forwards data, it performs concurrent forwarding in the Reactor multi-threaded mode, initializes an instance to listen to and distribute network events, and registers events for each QP. The main thread polls and receives CQ events. After receiving the message, it determines the forwarding target node according to the application layer routing table, allocates a thread for each forwarding connection from the thread pool, copies the message content to the sending buffer, and initiates an asynchronous operation. When performing a dual-end send-receive operation SEND-RECEIVE, the sender encapsulates the data into a data packet that can be transmitted on the link, and the receiver parses the data. When performing a single-end read-write operation READ-WRITE, after obtaining the destination address through the SEND-RECEIVE operation, the remote memory can be read and written directly without the need for the other end CPU to participate in perceiving the data being read and written in the memory.

[0023] S4. Interface call. Develop C and JAVA API interfaces respectively to implement a data transmission solution, support bilateral operations of SEND-RECEIVE, unilateral read READ and unilateral write WRITE operations of C and JAVA programs in multicast scenarios, and set the number of sent / received / read / written data, data packet size, and number of transmissions through parameters.

[0024] The specific embodiments are described as follows.

[0025] Example 1: Multicast group division The structure of some production scenarios of the simulation market system, such as Figure 2 The hierarchical multicast structure is constructed as shown, and the sending, forwarding and receiving nodes are set according to different types of components. The sending node S1 is responsible for generating and sending market information messages; the forwarding node F1 is responsible for receiving messages from the upper node and forwarding them to the lower node; the receiving nodes R1, R2, R3, and R4 are similar to the leaf nodes of the tree structure and are only responsible for receiving market data.

[0026] Every two layers of nodes form a partition area, each area contains multicast sources and multicast groups, and the multicast groups in each area are assigned different IP addresses. The forwarding node joins different partition areas as both a sender and a receiver, that is, as a receiver in the multicast group of the upper layer area and as a sender in the multicast source of the lower layer area. Specifically, the sending node S1 in partition area 1 is the multicast source, and the second-layer nodes R1, R2, and F1 are a multicast group. The forwarding node F1 in partition area 2 is the multicast source, and the second-layer nodes R3 and R4 are a multicast group. As the level deepens, other partition areas are similar.

[0027] Example 2: Control message sending and receiving and connection establishment Based on Ethernet and the above multicast structure, data sending and receiving operations are implemented. Socket communication is used to send control messages and establish connections based on the RDMA CM function library. In divided area 1, the sending node S1 uses the UDP protocol as an envelope, and the main thread encapsulates control information. The control information includes S1's IP address IP1 and port information Port1. The forwarding node F1 and the receiving nodes R1 and R2 listen to the UDP message and parse the message content after receiving it. In divided area 2, the forwarding node F1 acts as the sender to send a control UDP message, including F1's IP address IP2 and port information Port2. The members of multicast group 2 listen to and parse the message. Figure 3 An application layer routing table is created synchronously for subsequent data message transmission between nodes. The table entries include: multicast source, IP address and communication port of nodes in the multicast group.

[0028] In area 1, the receivers R1, R2, and F1 parse the address information and send connection requests. The sender S1 monitors the connection request of the receiver based on RC, and creates the communication endpoints EP of S1, R1, R2, and F1 in sequence according to the request processing order. For each pair of connections, a pair of communication endpoints EP must be established. The connection handshake process is processed by exchanging the memory MR addresses of both ends through the RDMA CM library, and a work queue pair QP is created on the EP, and the QP ready state transition is set. Similarly, in area 2, the forwarding node F1 needs to establish EPs corresponding to different connections according to the connection conditions, thereby realizing the multicast forwarding simulation scenario.

[0029] Example 3: C-interface data plane message For the data plane double-end sending and receiving operation SEND-RECEIVE, data is sent, forwarded and received through RC transmission based on the established connection. The working processes of the sending node, forwarding node and receiving node are as follows: Figure 4 , 5 , 6. The sending node starts multithreading to send control messages and data messages respectively, the forwarding node starts multithreading to complete data sending and receiving concurrently, and the receiving node receives data. The timing relationship is as follows Figure 7 shown.

[0030] Taking the SEND-RECEIVE operation as an example, after creating the QP, a send or receive event is registered for each QP. The data of the sender S1 in the divided area 1 is encapsulated into a data packet that can be transmitted on the link. After the transmission is based on the established connection, F1, as the receiving end, parses the data and stores the data in the specified memory area according to the work queue element WQE. It returns the ACK information and puts the completion information into the completion queue CQ. In particular, the forwarding node F1 performs concurrent forwarding in the Reactor multi-threaded mode. The main thread initializes an instance, which is responsible for listening to and distributing events. The instance cyclically listens to the reception completion CQ event, determines the forwarding target node information in combination with the application layer routing table, and allocates a working thread from the thread pool for each pair of data transmission on the forwarding connection. The forwarding function is called back in the receiving event, such as Figure 8 As shown, the message content is copied to the F1 forwarding EP and sent asynchronously to the multicast group 2 members through the SEND method. When performing a single-end read / write operation READ-WRITE, the destination address can be obtained through the SEND-RECEIVE operation, and the remote memory READ / WRITE operation can be continued in the same way, without the need for the remote CPU to participate in the data being read and written in the memory. Through this kind of point-to-point simulated multicast mode and the reliable transmission mode of the RC itself, reliable multicast communication can be achieved.

[0031] Example 4: JAVA interface call Encapsulate the developed C interface, add JNI function declarations for each encapsulated function, define JAVA classes and their native methods after compilation, further implement JNI methods, interact with C interfaces through JNI library loading and JAVA interface calls, implement SEND-RECEIVE and READ-WRITE operations of JAVA interfaces in multicast scenarios, and support setting parameters such as the number of sent / received / read / write data, data packet size, and number of sends.

[0032] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A reliable multicast transmission method for RoCE protocol, characterized in that: The method comprises S1. Multicast group division: According to the actual market forwarding scenario, multicast sources and multicast groups are divided in a hierarchical manner, and nodes are divided into sending nodes, forwarding nodes and receiving nodes. The sending node, as a multicast source, is only responsible for producing and sending data; the forwarding node is responsible for receiving messages from the sending node or other forwarding nodes and forwarding them to other forwarding nodes or receiving nodes; the receiving node, as the end of the multicast group, only aggregates and receives the received data; S2. Control plane: The sender sends control information in User Datagram Protocol (UDP), sends its own Internet Protocol (IP) and port to the receiver, and the receiver listens to control messages, synchronizes the application layer routing table, creates a communication endpoint (EP) for establishing a communication connection, establishes a connection through the application programming interface (API) of the communication manager (CM) in the remote direct memory access (RDMA) technology, and completes the ready state transition of the work queue to the QP. S3. Data plane: The sender and the receiver use the reliable connection mode RC mode for reliable data transmission. The forwarding node uses the reactor multi-threaded mode Reactor multi-threaded mode for concurrent forwarding. The main thread polls the receive completion queue CQ event, determines the forwarding target node according to the application layer routing table, copies the message content to the send buffer and initiates an asynchronous operation to perform the send-receive bilateral operation SEND-RECEIVE operation and the read-write unilateral operation READ-WRITE operation. S4. Interface call: Develop C and JAVA API interfaces to implement the send-receive bilateral operation SEND-RECEIVE operation and read-write unilateral operation READ-WRITE operation of C and JAVA programs in multicast scenarios, and set the number of data items sent or received or read or written, data packet size and number of transmission times through parameters.

2. A reliable multicast transmission method for RoCE protocol as claimed in claim 1, characterized in that: In the control plane step, the sending end includes a sending node and a forwarding node. The sending end uses the User Datagram Protocol UDP protocol as an envelope to send control information and sends its own Internet Protocol IP and port to the receiving end. The receiving end includes a receiving node and a forwarding node. The receiving end monitors control messages, synchronizes the application layer routing table, and sequentially creates a communication endpoint EP for establishing a communication connection. A connection processing handshake process is established based on the communication endpoint EP information and the application programming interface API, a work queue pair QP is created on the communication endpoint EP, the work queue pair QP is associated with the communication endpoint through the application programming interface API, and the work queue pair QP is completed. The ready state transition.

3. A reliable multicast transmission method for RoCE protocol as claimed in claim 1, characterized in that: In the data plane step, the sending end and the receiving end use the reliable connection mode RC mode to reliably transmit data. When the forwarding node forwards data, it uses the reactor multi-threaded mode Reactor multi-threaded mode for concurrent forwarding. An instance is initialized to monitor and distribute network events, register events for each work queue QP, and the main thread polls the reception completion queue CQ event. After receiving the message, the forwarding target node is determined according to the application layer routing table, and a thread is allocated from the thread pool for each forwarding connection. The message content is copied to the sending buffer and an asynchronous operation is initiated. When performing a double-end send-receive bilateral operation SEND-RECEIVE operation, the sending end data is encapsulated into a data packet that can be transmitted on the link, and the receiving end parses the data. When performing a single-end read-write unilateral operation READ-WRITE operation, after obtaining the destination address through the send-receive bilateral operation SEND-RECEIVE operation, the remote memory can be directly read and written without the remote CPU participating in sensing the data being read and written in the memory.

4. A reliable multicast transmission method for RoCE protocol as claimed in claim 1, characterized in that: In the interface calling step, C and JAVA API interface development are performed respectively to implement a set of data transmission solutions, supporting C and JAVA programs to send and receive send-receive bilateral operations SEND-RECEIVE operations, read-write unilateral operations READ-WRITE operations of unilateral read READ and unilateral write WRITE operations in multicast scenarios, and setting the number of data items sent or received or read or written, the size of the data packet and the number of transmission times through parameters.

Images