DDS message low-latency transmission method based on RoCE network

Abstract

The present application relates to a kind of DDS message low latency transmission method based on RoCE network, the technical problem to be solved is how to run existing DDS software in RoCE network, realize the low latency transmission of DDS message by low latency RoCE network, existing DDS software bottom layer communication is based on TCP / IP protocol, the socket interface used in the interface of development is incompatible with the interface verbs of the interface of development of RoCE network, so that existing DDS software cannot run in RoCE network, the present application proposes a kind of DDS message low latency transmission method based on RoCE network, this method uses network communication middleware technology, realizes the seamless connection of socket interface of TCP / IP network and the interface verbs of the interface of development of RoCE network, so that existing DDS software can be directly run in RoCE network without modifying code, reduce the end-to-end transmission delay of DDS message by the low latency characteristics of RoCE network.

Description

Technical Field

[0001] This invention belongs to the field of computer network technology, and particularly relates to a low-latency transmission method for data distribution service (DDS) messages based on RoCE (RDMA over Coverage Ethernet) network. Background Technology

[0002] Data Storage and Automation (DDS) is a data-centric middleware protocol and API standard released by the Object Management Group (OMG). DDS was initially used in the US Navy to address compatibility issues arising from numerous software upgrades in the complex network environment of warship systems. Currently, DDS is primarily used in fields such as autonomous driving and robotics. Around 2015, domestic researchers began to widely study the application of DDS technology in shipborne, vehicle-mounted, and airborne equipment computing platforms. Currently, DDS technology is widely used in a large number of high-end equipment systems, becoming the basic architecture for inter-service communication between high-end equipment platforms. Solving the problems of real-time performance and reliability in data distribution between high-end equipment services hinges on DDS technology.

[0003] With the development of artificial intelligence technology, unmanned and intelligent operation has become a significant trend in the future development of high-end equipment. This places high demands on the real-time performance and reliability of equipment operation software. How to achieve low-latency transmission of DDS messages is a problem that DDS technology must solve in the era of artificial intelligence.

[0004] Existing DDS protocol implementations, including OpenDDS and FastDDS, are all based on Ethernet and TCP / UDP protocols. Due to their network and protocol stack architectures, Ethernet and TCP / UDP networks require high terminal computing power and have significant end-to-end latency. The transmission latency of TCP / UDP networks determines the lower limit of message transmission latency for existing DDS technologies.

[0005] RoCE is an RDMA (Remote Direct Memory Access) network implemented on top of the Ethernet IP / UDP protocol. Its principle is based on InfiniBand (the main implementation of RDMA networks, with the protocol specification InfiniBand). TMThe transport layer messages of the RoCE architecture specification (Volume 1, Release 1.4) are encapsulated in the payload of UDP packets, and InfiniBand messages are transmitted via the Ethernet IP / UDP protocol. This allows for the implementation of RDMA networks in upper-layer protocols using InfiniBand (see the RoCE network protocol specification, Supplement to InfiniBand™ Architecture Specification Volume 1, Release 1.2.1, Annex A17: RoCEv2). Due to its low end-to-end latency, low CPU resource consumption, and low deployment cost, RoCE networks are widely deployed and used in data center networks for applications with high requirements for network latency and CPU utilization, such as AI and supercomputing.

[0006] Replacing TCP / IP with RoCE can significantly reduce end-to-end network transmission latency, thereby reducing DDS message transmission latency. However, because RoCE differs from TCP / IP (socket interface) in its application development interface (verbs interface), RoCE-based applications require a dedicated programming interface (see RDMA Network Programming User Manual Release 1.4). Existing DDS software cannot run directly on RoCE. Summary of the Invention

[0007] The technical problem this invention aims to solve is how to run existing DDS software on a RoCE network, achieving low-latency transmission of DDS messages through the low-latency RoCE network. Addressing the issue that existing DDS software, based on the TCP / IP protocol and using a socket interface for its development interface, is incompatible with the RoCE network's verbs interface, thus preventing its operation on RoCE networks, this invention proposes a low-latency DDS message transmission method based on RoCE networks. This method utilizes network communication middleware technology to achieve seamless integration between the TCP / IP network's socket interface and the RoCE network's verbs interface, allowing existing DDS software to run directly on RoCE networks without code modification. By leveraging the low-latency characteristics of the RoCE network, the end-to-end transmission latency of DDS messages is reduced.

[0008] The technical solution adopted in this invention is a low-latency transmission method for DDS messages based on RoCE networks, which consists of the following steps: The first step is to construct a low-latency DDS message transmission system based on the RoCE network. This system consists of an initialization module, a connection parameter acquisition module, a connection management data structure, a connection management module, a data sending module, a data sending buffer, a data receiving module, and a data receiving buffer.

[0009] The initialization module is used for initialization during system loading, including initializing the data sending buffer, the data receiving buffer, and the connection management data structure.

[0010] The data send buffer acts as a relay station between the socket interface for sending data and the RoCE network for transmitting data. When the DDS software calls the socket interface's sendto function to send data, the data is copied from the DDS software's memory to the data send buffer, and then transmitted to the receiving end via the RoCE network. Because TCP packets are generally no larger than 64KB, the data send buffer size is set to 64KB.

[0011] The data receive buffer acts as a relay station between data received via the socket interface and data transmitted over the RoCE network. When the DDS software calls the socket interface `recv` to receive data, the data is received by the receiving end via the RoCE network and stored in the data receive buffer. Then, the data receiving module copies the data from the data receive buffer to the DDS software's data buffer, completing the data reception. Because TCP packets are generally no larger than 64KB, the data receive buffer size is set to 64KB.

[0012] The connection management data structure records the mapping relationship between TCP connections and QP connections, as well as the working parameters of TCP and QP connections. Specific parameters include: the socket descriptor s_id identifying the TCP connection; the source IP address s_ip, destination IP address d_ip, source port number s_port, and destination port number d_port identifying the TCP connection; the QP (QueuePair, the meaning of which is described in Supplement to InfiniBand™ Architecture Specification Volume 1 Release 1.2.1 Annex A17: RoCEv2) connection qp; and the CQ (CompletionQueue) queue cq used by the QP connection. (The data structures for QP and CQ are described in the RDMA Adware Network Programming User Manual Release.) 1.4 and the open-source project https: / / github.com / linux-rdma / rdma-core.git, subsequent data structures such as QP, CQ, MR, and WR will all refer to these two documents), QP number qpn, data send buffer s_mr used by QP connection, data receive buffer r_mr, memory address of r_mr r_buf_addr, descriptor tcp_cnn indicating whether a TCP connection has been established (tcp_cnn is 1, indicating a TCP connection has been established; tcp_cnn is 0, indicating a TCP connection has not been established), descriptor qp_cnn indicating whether a QP connection has been established (qp_cnn is 1, indicating a QP connection has been established; qp_cnn is 0, indicating a QP connection has not been established), QP number rqpn of the peer connection, address of the peer connection's receive buffer raddr, and access key rkey of the peer connection's receive buffer.

[0013] The connection parameter acquisition module is used to monitor the system calls of the DDS software, and obtain various parameters required for establishing a QP connection in the RoCE network from system calls such as socket, connect, bind, listen, and accept, and update the connection management data structure.

[0014] The connection management module is used to obtain data from the connection management data structure obtained by the connection parameter acquisition module, interact with the RoCE network protocol stack according to the RDMA Adware Network Programming User Manual Release 1.4, establish a QP connection for data transmission in the RoCE network, and update the connection management data structure.

[0015] The data transmission module monitors the system calls made by the DDS software to send data, interacts with the RoCE network protocol stack, and uses the RoCE network's data transmission method instead of the TCP / IP network's data transmission method to complete the data transmission. When the DDS software calls the sendto interface to send data over the TCP / IP network, the data transmission module retrieves the data to be sent from the DDS software, stores it in the data transmission buffer, and interacts with the RoCE network protocol stack using the QP connection in the connection management data structure to complete the data transmission.

[0016] The data receiving module monitors the system calls of the DDS software to receive data and interacts with the RoCE network protocol stack, using the RoCE network data receiving method instead of the TCP / IP network data receiving method to complete data reception. When the DDS software calls the recv interface to receive data through the TCP / IP network, the data receiving module uses the QP connection in the connection management data structure to interact with the RoCE network protocol stack to complete data reception and deliver the received data to the DDS software through the data receiving buffer.

[0017] The second step is for the initialization module to complete system initialization.

[0018] 2.1 Initialize the connection management data buffer The connection management data buffer is used to store connection management data structures. These data structures record the operating parameters and status of TCP and QP connections, as well as the mapping relationship between TCP and QP connections.

[0019] The method for initializing the connection management data buffer is as follows: 2.1.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 4KB buffer as a connection management data buffer to store the working parameters of the connection management data structure.

[0020] 2.1.2 Clear all data in the connection management data buffer.

[0021] 2.2 Initialize the data transmission buffer The method for initializing the data transmission buffer in the initialization module is as follows: 2.2.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 64K byte buffer as a data transmission buffer and obtain the memory address of the buffer; 2.2.2 Using the buffer memory address and size (64K) obtained in step 2.2.1, create an MR for the data send buffer using the MR creation interface in the verbbs interface (ibv_reg_mr, etc., see RDMA Adware Network Programming User Manual Release 1.4 and the open source project https: / / github.com / linux-rdma / rdma-core.git, all subsequent use of the verbbs interface will refer to these two documents) named send_mr.

[0022] 2.2.3 Update the connection management data structure, setting s_mr=send_mr.

[0023] 2.3 Initialize the data receive buffer 2.3.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 64K byte buffer as a data receiving buffer and obtain the memory address of the buffer; 2.3.2 Using the buffer memory address and size (64K) obtained in step 2.2.1, create an MR for the data transmission buffer using the Create MR interface (ibv_reg_mr, etc.) in the verbs interface. The MR is named recv_mr.

[0024] 2.3.3 Update the connection management data structure, setting r_mr=recv_mr.

[0025] The third step is to obtain the connection parameters of the DDS software using the connection parameter acquisition module.

[0026] The connection parameter acquisition module monitors the system calls of the DDS software. When the DDS software calls interfaces related to establishing a TCP connection, the connection parameter acquisition module obtains the input or output parameters of the system call and updates the connection management data structure. The specific steps are as follows: 3.1 The connection parameter acquisition module monitors the system calls of the DDS software. When the DDS software calls socket, proceed to 3.2; when the DDS software calls connect, proceed to 3.3; when the DDS software calls bind, proceed to 3.4; when the DDS software calls accept, proceed to 3.5; otherwise, continue monitoring the system calls of the DDS software.

[0027] 3.2 The connection parameter acquisition module obtains the socket's return value and updates the connection management data structure. The specific steps are as follows: 3.2.1 Continue executing the socket system call and obtain the return value socket_id; 3.2.2 If socket_id is greater than 0, it means the socket execution is correct. Update the connection management data structure, set s_id = socket_id, and go to 3.1; otherwise, it means the socket execution is wrong. Do not update the connection management data structure, and go directly to 3.1.

[0028] 3.3 Obtain the input parameters and return value of connect and update the connection management data structure. The specific steps are as follows: 3.3.1 Obtain the IP address and port number from the input parameters of connect, and let the IP address be dest_IP and the port number be dest_port; 3.3.2 Continue executing connect and obtain the return value ret_connect; 3.3.3 If ret_connect equals 0, it means connect was executed correctly. Update the connection management data structure, set d_ip=dest_IP, set d_port=dest_port, and go to 3.1; otherwise, it means connect was executed incorrectly. Do not update the connection management data structure, and go directly to 3.1.

[0029] 3.4 Obtain the input parameters and return value of bind and update the connection management data structure. The specific steps are as follows: 3.4.1 Obtain the IP address and port number from the input parameters of bind, and let the IP address be src_IP and the port number be src_port; 3.4.2 Continue executing bind and obtain the return value ret_bind; 3.4.3 If ret_bind equals 0, it means that bind was executed correctly. Update the connection management data structure, set s_ip=src_IP, set s_port=src_port, and go to 3.1; otherwise, it means that bind was executed incorrectly. Do not update the connection management data structure, and go directly to 3.1.

[0030] 3.5 Obtain the input and output parameters of accept and update the connection management data structure. The specific steps are as follows: 3.5.1 Continue executing accept and obtain the return value ret_accept; 3.5.2 Obtain the IP address and port number from the accept output parameters, and let the IP address be rem_ip and the port number be rem_port; 3.5.3 If ret_accept equals 0, it means that accept was executed correctly. Update the connection management data structure, set d_ip=rem_ip, set d_port=rem_port, and set tcp_cnn=1.

[0031] The fourth step involves the connection management module establishing a QP connection associated with the TCP connection based on the connection management data structure, in order to use the RoCE network to transmit DDS messages.

[0032] The specific steps are as follows: 4.1 Create the CQ queue required for the QP connection and update the connection management data structure. The specific steps are as follows: 4.1.1 Call the verbs interface for creating a CQ (such as ibv_create_cq) to create the CQ queue l_cq; 4.1.2 Update the connection management data structure, and set cq=l_cq.

[0033] 4.2 Create a QP and update the connection management data structure. The specific steps are as follows: 4.2.1 Call the verbs interface for creating QPs (such as ibv_create_qp) to create the QP queue l_qp; 4.2.2 Update the connection management data structure, and set qp=l_qp.

[0034] 4.3 Exchange QP connection data and update the connection management data structure.

[0035] The two parties in the communication are described as the local end and the remote end, respectively.

[0036] The specific steps are as follows: 4.3.1 Construct the QP connection data structure, including "local qpn" (i.e., qpn in the local connection management data structure), "local receive buffer memory starting address" (i.e., r_buf_addr in the local connection management data structure), and "local receive buffer remote access key" (i.e., rkey field of r_mr in the local connection management data structure). 4.3.2 Using parameters such as s_id, d_ip, and d_port, the QP connection data structure is sent to the remote end via a TCP connection; 4.3.3 Receiving QP connection data structure sent from the remote end via TCP connection; 4.3.4 Update the "local qpn" in the QP connection data structure sent from the remote end to the rqpn in the connection management data structure, update the "local receive buffer memory start address" in it to the raddr in the connection management data structure, and update the "local receive buffer remote access key" in it to the rkey in the connection management data structure; 4.3.5 Using qp, rqpn, raddr, and rkey from the connection management data structure as parameters, call the verbs interface (such as ibv_modify_qp) to switch the QP state to a sendable and receiveable state.

[0037] Fifth, the data sending module sends DDS messages using a QP connection based on the RoCE network.

[0038] The data transmission module monitors the system calls of the DDS software. When the DDS software uses the socket interface to send data, it intercepts the data to be sent and uses the QP connection and connection management data structure established in step four to send the data to the receiving end via the RoCE network. The specific steps are as follows: 5.1 Monitor the system calls of the DDS software. When the DDS software calls the sendto or write interface, proceed to 5.2; otherwise, continue monitoring the system calls of the DDS software.

[0039] 5.2 Intercept the sendto or write system call, obtain the data send buffer address send_addr and size send_len from the input parameters, and prevent the execution of the system call; 5.3 Data of size send_len, starting from address send_addr, is copied to the buffer of s_mr; 5.4 Construct the WR data structure s_wr for sending work requests using s_mr, rqpn, raddr, and rkey as parameters; 5.5 Using qp and s_wr as parameters, call the verbs interface (such as ibv_post_send) to send the work request to the RoCE protocol stack; 5.6 Return to system call, go to 5.1 to continue monitoring the system calls of the DDS software.

[0040] Step 6: The data receiving module receives DDS messages using a QP connection based on the RoCE network.

[0041] The data receiving module monitors the system calls of the DDS software. When the DDS software uses the socket interface to receive data, it uses the QP connection and connection management data structure established in step four to complete data reception based on the RoCE network. The specific steps are as follows: 6.1 Monitor the system calls of the DDS software. When the DDS software calls recv or read, proceed to 6.2; otherwise, continue monitoring the system calls of the DDS software.

[0042] 6.2 Intercept the recv or read system call, obtain the data receive buffer address recv_addr and size recv_len from the input parameters, and prevent the execution of the system call; 6.3 Copy the first recv_len bytes of data in the r_mr buffer to the memory address recv_addr; 6.4 Return to system call, go to 6.1 to continue monitoring the system calls of the DDS software.

[0043] The following technical effects can be achieved by using this invention: 1) The first step of this invention utilizes the technical features of the RoCE network user programming interface to design a low-latency DDS message transmission system based on the RoCE network. This system acts as a network communication middleware and uses the RoCE network user programming interface to implement the functions of the TCP network, enabling the transmission of DDS messages that were originally based on the TCP network using the RoCE network.

[0044] 2) The third step of this invention constructs a connection management data structure based on the parameters for establishing a TCP connection between the two ends of the communication. The fourth step establishes a QP connection between the two ends based on the connection management data structure, thereby establishing a mapping relationship between QP connections and TCP connections in the RoCE network, providing a foundation for transmitting TCP data using the RoCE network.

[0045] 3) In steps five and six of this invention, the execution of TCP network data transmission and reception system calls is intercepted and blocked. At the same time, the transmission of DDS messages is realized by using QP connection based on RoCE network. The low latency characteristics of RoCE network transmission reduce the transmission latency of DDS messages.

[0046] 4) This invention can be directly used for message transmission in DDS software, and with appropriate modifications to the application background, it can also be used for data transmission in other TCP network-based applications. Attached Figure Description

[0047] Figure 1 This is the overall flowchart of the present invention; Figure 2 This is a logical structure diagram of the low-latency DDS message transmission system based on RoCE network constructed in the first step of this invention. Detailed Implementation

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0049] Figure 1 This is the overall flowchart of the present invention, a low-latency transmission method for DDS messages based on RoCE networks, which consists of the following steps: The first step is to construct a low-latency DDS message transmission system based on a RoCE network. This system consists of an initialization module, a connection parameter acquisition module, a connection management data structure, a connection management module, a data sending module, a data sending buffer, a data receiving module, and a data receiving buffer, as follows: Figure 2 As shown.

[0050] The initialization module is used for initialization during system loading, including initializing the data sending buffer, the data receiving buffer, and the connection management data structure.

[0051] The data send buffer acts as a relay station between the socket interface for sending data and the RoCE network for transmitting data. When the DDS software calls the socket interface's sendto function to send data, the data is copied from the DDS software's memory to the data send buffer, and then transmitted to the receiving end via the RoCE network. Because TCP packets are generally no larger than 64KB, the data send buffer size is set to 64KB.

[0052] The data receive buffer acts as a relay station between data received via the socket interface and data transmitted over the RoCE network. When the DDS software calls the socket interface `recv` to receive data, the data is received by the receiving end via the RoCE network and stored in the data receive buffer. Then, the data receiving module copies the data from the data receive buffer to the DDS software's data buffer, completing the data reception. Because TCP packets are generally no larger than 64KB, the data receive buffer size is set to 64KB.

[0053] The connection management data structure records the mapping relationship between TCP connections and QP connections, as well as the working parameters of TCP and QP connections. Specific parameters include: the socket descriptor s_id identifying the TCP connection; the source IP address s_ip, destination IP address d_ip, source port number s_port, and destination port number d_port identifying the TCP connection; the QP (QueuePair, the meaning of which is described in Supplement to InfiniBand™ Architecture Specification Volume 1 Release 1.2.1 Annex A17: RoCEv2) connection qp; and the CQ (CompletionQueue) queue cq used by the QP connection. (The data structures for QP and CQ are described in the RDMA Adware Network Programming User Manual Release.) 1.4 and the open-source project https: / / github.com / linux-rdma / rdma-core.git, subsequent data structures such as QP, CQ, MR, and WR will all refer to these two documents), QP number qpn, data send buffer s_mr used by QP connection, data receive buffer r_mr, memory address of r_mr r_buf_addr, descriptor tcp_cnn indicating whether a TCP connection has been established (tcp_cnn is 1, indicating a TCP connection has been established; tcp_cnn is 0, indicating a TCP connection has not been established), descriptor qp_cnn indicating whether a QP connection has been established (qp_cnn is 1, indicating a QP connection has been established; qp_cnn is 0, indicating a QP connection has not been established), QP number rqpn of the peer connection, address of the peer connection's receive buffer raddr, and access key rkey of the peer connection's receive buffer.

[0054] The connection parameter acquisition module is used to monitor the system calls of the DDS software, and obtain various parameters required for establishing a QP connection in the RoCE network from system calls such as socket, connect, bind, listen, and accept, and update the connection management data structure.

[0055] The connection management module is used to obtain data from the connection management data structure obtained by the connection parameter acquisition module, interact with the RoCE network protocol stack according to the RDMA Adware Network Programming User Manual Release 1.4, establish a QP connection for data transmission in the RoCE network, and update the connection management data structure.

[0056] The data transmission module monitors the system calls made by the DDS software to send data, interacts with the RoCE network protocol stack, and uses the RoCE network's data transmission method instead of the TCP / IP network's data transmission method to complete the data transmission. When the DDS software calls the sendto interface to send data over the TCP / IP network, the data transmission module retrieves the data to be sent from the DDS software, stores it in the data transmission buffer, and interacts with the RoCE network protocol stack using the QP connection in the connection management data structure to complete the data transmission.

[0057] The data receiving module monitors the system calls of the DDS software to receive data and interacts with the RoCE network protocol stack, using the RoCE network data receiving method instead of the TCP / IP network data receiving method to complete data reception. When the DDS software calls the recv interface to receive data through the TCP / IP network, the data receiving module uses the QP connection in the connection management data structure to interact with the RoCE network protocol stack to complete data reception and deliver the received data to the DDS software through the data receiving buffer.

[0058] The second step is for the initialization module to complete system initialization.

[0059] 2.1 Initialize the connection management data buffer The connection management data buffer is used to store connection management data structures. These data structures record the operating parameters and status of TCP and QP connections, as well as the mapping relationship between TCP and QP connections.

[0060] The method for initializing the connection management data buffer is as follows: 2.1.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 4KB buffer as a connection management data buffer to store the working parameters of the connection management data structure.

[0061] 2.1.2 Clear all data in the connection management data buffer.

[0062] 2.2 Initialize the data transmission buffer The method for initializing the data transmission buffer in the initialization module is as follows: 2.2.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 64K byte buffer as a data transmission buffer and obtain the memory address of the buffer; 2.2.2 Using the buffer memory address and size (64K) obtained in step 2.2.1, create an MR for the data send buffer using the MR creation interface in the verbbs interface (ibv_reg_mr, etc., see RDMA Adware Network Programming User Manual Release 1.4 and the open source project https: / / github.com / linux-rdma / rdma-core.git, all subsequent use of the verbbs interface will refer to these two documents) named send_mr.

[0063] 2.2.3 Update the connection management data structure, setting s_mr=send_mr.

[0064] 2.3 Initialize the data receive buffer 2.3.1 Use the operating system's memory allocation interface (malloc, calloc, etc.) to allocate a 64K byte buffer as a data receiving buffer and obtain the memory address of the buffer; 2.3.2 Using the buffer memory address and size (64K) obtained in step 2.2.1, create an MR for the data transmission buffer using the Create MR interface (ibv_reg_mr, etc.) in the verbs interface. The MR is named recv_mr.

[0065] 2.3.3 Update the connection management data structure, setting r_mr=recv_mr.

[0066] The third step is to obtain the connection parameters of the DDS software using the connection parameter acquisition module.

[0067] The connection parameter acquisition module monitors the system calls of the DDS software. When the DDS software calls interfaces related to establishing a TCP connection, the connection parameter acquisition module obtains the input or output parameters of the system call and updates the connection management data structure. The specific steps are as follows: 3.1 The connection parameter acquisition module monitors the system calls of the DDS software. When the DDS software calls socket, proceed to 3.2; when the DDS software calls connect, proceed to 3.3; when the DDS software calls bind, proceed to 3.4; when the DDS software calls accept, proceed to 3.5; otherwise, continue monitoring the system calls of the DDS software.

[0068] 3.2 The connection parameter acquisition module obtains the socket's return value and updates the connection management data structure. The specific steps are as follows: 3.2.1 Continue executing the socket system call and obtain the return value socket_id; 3.2.2 If socket_id is greater than 0, it means the socket execution is correct. Update the connection management data structure, set s_id = socket_id, and go to 3.1; otherwise, it means the socket execution is wrong. Do not update the connection management data structure, and go directly to 3.1.

[0069] 3.3 Obtain the input parameters and return value of connect and update the connection management data structure. The specific steps are as follows: 3.3.1 Obtain the IP address and port number from the input parameters of connect, and let the IP address be dest_IP and the port number be dest_port; 3.3.2 Continue executing connect and obtain the return value ret_connect; 3.3.3 If ret_connect equals 0, it means connect was executed correctly. Update the connection management data structure, set d_ip=dest_IP, set d_port=dest_port, and go to 3.1; otherwise, it means connect was executed incorrectly. Do not update the connection management data structure, and go directly to 3.1.

[0070] 3.4 Obtain the input parameters and return value of bind and update the connection management data structure. The specific steps are as follows: 3.4.1 Obtain the IP address and port number from the input parameters of bind, and let the IP address be src_IP and the port number be src_port; 3.4.2 Continue executing bind and obtain the return value ret_bind; 3.4.3 If ret_bind equals 0, it means that bind was executed correctly. Update the connection management data structure, set s_ip=src_IP, set s_port=src_port, and go to 3.1; otherwise, it means that bind was executed incorrectly. Do not update the connection management data structure, and go directly to 3.1.

[0071] 3.5 Obtain the input and output parameters of accept and update the connection management data structure. The specific steps are as follows: 3.5.1 Continue executing accept and obtain the return value ret_accept; 3.5.2 Obtain the IP address and port number from the accept output parameters, and let the IP address be rem_ip and the port number be rem_port; 3.5.3 If ret_accept equals 0, it means that accept was executed correctly. Update the connection management data structure, set d_ip=rem_ip, set d_port=rem_port, and set tcp_cnn=1.

[0072] The fourth step involves the connection management module establishing a QP connection associated with the TCP connection based on the connection management data structure, in order to use the RoCE network to transmit DDS messages.

[0073] The specific steps are as follows: 4.1 Create the CQ queue required for the QP connection and update the connection management data structure. The specific steps are as follows: 4.1.1 Call the verbs interface for creating a CQ (such as ibv_create_cq) to create the CQ queue l_cq; 4.1.2 Update the connection management data structure, and set cq=l_cq.

[0074] 4.2 Create a QP and update the connection management data structure. The specific steps are as follows: 4.2.1 Call the verbs interface for creating QPs (such as ibv_create_qp) to create the QP queue l_qp; 4.2.2 Update the connection management data structure, and set qp=l_qp.

[0075] 4.3 Exchange QP connection data and update the connection management data structure.

[0076] The two parties in the communication are described as the local end and the remote end, respectively.

[0077] The specific steps are as follows: 4.3.1 Construct the QP connection data structure, including "local qpn" (i.e., qpn in the local connection management data structure), "local receive buffer memory starting address" (i.e., r_buf_addr in the local connection management data structure), and "local receive buffer remote access key" (i.e., rkey field of r_mr in the local connection management data structure). 4.3.2 Using parameters such as s_id, d_ip, and d_port, the QP connection data structure is sent to the remote end via a TCP connection; 4.3.3 Receiving QP connection data structure sent from the remote end via TCP connection; 4.3.4 Update the "local qpn" in the QP connection data structure sent from the remote end to the rqpn in the connection management data structure, update the "local receive buffer memory start address" in it to the raddr in the connection management data structure, and update the "local receive buffer remote access key" in it to the rkey in the connection management data structure; 4.3.5 Using qp, rqpn, raddr, and rkey from the connection management data structure as parameters, call the verbs interface (such as ibv_modify_qp) to switch the QP state to a sendable and receiveable state.

[0078] Fifth, the data sending module sends DDS messages using a QP connection based on the RoCE network.

[0079] The data transmission module monitors the system calls of the DDS software. When the DDS software uses the socket interface to send data, it intercepts the data to be sent and uses the QP connection and connection management data structure established in step four to send the data to the receiving end via the RoCE network. The specific steps are as follows: 5.1 Monitor the system calls of the DDS software. When the DDS software calls the sendto or write interface, proceed to 5.2; otherwise, continue monitoring the system calls of the DDS software.

[0080] 5.2 Intercept the sendto or write system call, obtain the data send buffer address send_addr and size send_len from the input parameters, and prevent the execution of the system call; 5.3 Data of size send_len, starting from address send_addr, is copied to the buffer of s_mr; 5.4 Construct the WR data structure s_wr for sending work requests using s_mr, rqpn, raddr, and rkey as parameters; 5.5 Using qp and s_wr as parameters, call the verbs interface (such as ibv_post_send) to send the work request to the RoCE protocol stack; 5.6 Return to system call, go to 5.1 to continue monitoring the system calls of the DDS software.

[0081] Step 6: The data receiving module receives DDS messages using a QP connection based on the RoCE network.

[0082] The data receiving module monitors the system calls of the DDS software. When the DDS software uses the socket interface to receive data, it uses the QP connection and connection management data structure established in step four to complete data reception based on the RoCE network. The specific steps are as follows: 6.1 Monitor the system calls of the DDS software. When the DDS software calls recv or read, proceed to 6.2; otherwise, continue monitoring the system calls of the DDS software.

[0083] 6.2 Intercept the recv or read system call, obtain the data receive buffer address recv_addr and size recv_len from the input parameters, and prevent the execution of the system call; 6.3 Copy the first recv_len bytes of data in the r_mr buffer to the memory address recv_addr; 6.4 Return to system call, go to 6.1 to continue monitoring the system calls of the DDS software.

Claims

1. A method for low-latency transmission of DDS messages based on RoCE networks, characterized in that, This method consists of the following steps: The first step is to construct a low-latency DDS message transmission system based on a RoCE network. This system consists of an initialization module, a connection parameter acquisition module, a connection management data structure, a connection management module, a data sending module, a data sending buffer, a data receiving module, and a data receiving buffer. composition: The initialization module is used for system initialization during loading, including initializing the data sending buffer, initializing the data receiving buffer, and initializing the connection management data structure; The data sending buffer is the relay station between the socket interface sending data and the RoCE network transmitting data. When the DDS software calls the socket interface sendto to send data, the data is copied from the DDS software memory to the data sending buffer, and then transmitted to the receiving end by the RoCE network. The data receive buffer is the relay station between the socket interface receiving data and the RoCE network transmitting data. When the DDS software calls the socket interface recv to receive data, the data is received by the receiving end through the RoCE network and stored in the data receive buffer. Then the data receiving module copies the data from the data receive buffer to the DDS software data buffer to complete the data reception. The connection management data structure is used to record the mapping relationship between TCP connections and QP connections, as well as the working parameters of TCP connections and QP connections; The connection parameter acquisition module is used to monitor the system calls of the DDS software, and obtain various parameters required for establishing a QP connection in the RoCE network from the socket, connect, bind, listen, and accept system calls, and update the connection management data structure. The connection management module is used to obtain data from the connection management data structure obtained by the connection parameter acquisition module, interact with the RoCE network protocol stack according to the RDMA network programming user manual, establish a QP connection for data transmission in the RoCE network, and update the connection management data structure. The data sending module monitors the system calls of the DDS software to send data, interacts with the RoCE network protocol stack, and uses the RoCE network data sending method instead of the TCP / IP network data sending method to complete the data sending. When the DDS software calls the sendto interface to send data through the TCP / IP network, the data sending module obtains the data to be sent from the DDS software, stores it in the data sending buffer, and interacts with the RoCE network protocol stack using the QP connection in the connection management data structure to complete the data sending. The data receiving module monitors the system calls of the DDS software to receive data and interacts with the RoCE network protocol stack. It uses the RoCE network data receiving method instead of the TCP / IP network data receiving method to complete the data reception. When the DDS software calls the recv interface to receive data through the TCP / IP network, the data receiving module uses the QP connection in the connection management data structure to interact with the RoCE network protocol stack to complete the data reception and deliver the received data to the DDS software through the data receiving buffer. The second step involves the initialization module completing system initialization. 2.1 Initialize the connection management data buffer The connection management data buffer is used to store connection management data structures; The method for initializing the connection management data buffer is as follows: 2.1.1 Use the operating system's memory allocation interface to allocate a buffer as a connection management data buffer to store the working parameters of the connection management data structure; 2.1.2 Clear all data in the connection management data buffer; 2.2 Initialize the data transmission buffer The method for initializing the data transmission buffer in the initialization module is as follows: 2.2.1 Use the operating system's memory allocation interface to allocate a 64KB buffer as a data transmission buffer and obtain the memory address of the buffer; 2.2.2 Using the buffer memory address and size obtained in step 2.2.1, create an MR for the data sending buffer using the Create MR interface in the verbs interface, named send_mr; 2.2.3 Update the connection management data structure, setting s_mr = send_mr; 2.3 Initialize the data receive buffer 2.3.1 Use the operating system's memory allocation interface to allocate a 64KB buffer as a data receiving buffer and obtain the buffer's memory address; 2.3.2 Using the buffer memory address and size obtained in step 2.2.1, create an MR for the data transmission buffer using the Create MR interface in the verbs interface, named recv_mr; 2.3.3 Update the connection management data structure, setting r_mr = recv_mr; The third step is to obtain the connection parameters of the DDS software using the connection parameter acquisition module. The connection parameter acquisition module monitors the system calls of the DDS software. When the DDS software calls interfaces related to establishing a TCP connection, the connection parameter acquisition module obtains the input or output parameters of the system call and updates the connection management data structure. The specific steps are as follows: 3.1 The connection parameter acquisition module monitors the system calls of the DDS software; when the DDS software calls socket, proceed to 3.2; when the DDS software calls connect, proceed to 3.3; when the DDS software calls bind, proceed to 3.4; when the DDS software calls accept, proceed to 3.5; otherwise, continue monitoring the system calls of the DDS software. 3.2 The connection parameter acquisition module obtains the socket's return value and updates the connection management data structure; the specific steps are as follows: 3.2.1 Continue executing the socket system call and obtain the return value socket_id; 3.2.2 If socket_id is greater than 0, it means the socket execution is correct. Update the connection management data structure, set s_id = socket_id, and go to 3.1; otherwise, it means the socket execution is incorrect. Do not update the connection management data structure, and go directly to 3.

1. 3.3 Obtain the input parameters and return value of connect and update the connection management data structure; the specific steps are as follows: 3.3.1 Obtain the IP address and port number from the input parameters of connect, and let the IP address be dest_IP and the port number be dest_port; 3.3.2 Continue executing connect and obtain the return value ret_connect; 3.3.3 If ret_connect equals 0, it means connect executed correctly. Update the connection management data structure, set d_ip=dest_IP, set d_port=dest_port, and go to 3.1; otherwise, it means connect executed incorrectly. Do not update the connection management data structure, and go directly to 3.

1. 3.4 Obtain the input parameters and return value of bind and update the connection management data structure; the specific steps are as follows: 3.4.1 Obtain the IP address and port number from the input parameters of bind, and let the IP address be src_IP and the port number be src_port; 3.4.2 Continue executing bind and obtain the return value ret_bind; 3.4.3 If ret_bind equals 0, it means bind executed correctly. Update the connection management data structure, set s_ip=src_IP, set s_port=src_port, and go to 3.1; otherwise, it means bind executed incorrectly. Do not update the connection management data structure, and go directly to 3.

1. 3.5 Obtain the input and output parameters of accept and update the connection management data structure; the specific steps are as follows: 3.5.1 Continue executing accept and obtain the return value ret_accept; 3.5.2 Obtain the IP address and port number from the accept output parameters, and let the IP address be rem_ip and the port number be rem_port; 3.5.3 If ret_accept equals 0, it means that accept was executed correctly. Update the connection management data structure, set d_ip=rem_ip, set d_port=rem_port, and set tcp_cnn=1; The fourth step is for the connection management module to establish a QP connection associated with the TCP connection based on the connection management data structure, so as to use the RoCE network to transmit DDS messages. The specific steps are as follows: 4.1 Create the CQ queue required for the QP connection and update the connection management data structure; the specific steps are as follows: 4.1.1 Call the verbs interface for creating a CQ to create a CQ queue l_cq; 4.1.2 Update the connection management data structure, setting cq = l_cq; 4.2 Create a QP and update the connection management data structure; the specific steps are as follows: 4.2.1 Call the verbs interface for creating QP to create the QP queue l_qp; 4.2.2 Update the connection management data structure, setting qp = l_qp; 4.3 Exchange QP connection data and update the connection management data structure; The two parties in the communication are described as the local end and the remote end, respectively; The specific steps are as follows: 4.3.1 Construct the QP connection data structure, including "local qpn", "local receive buffer memory starting address", and "local receive buffer remote access key"; 4.3.2 Using parameters such as s_id, d_ip, and d_port, the QP connection data structure is sent to the remote end via a TCP connection; 4.3.3 Receiving QP connection data structure sent from the remote end via TCP connection; 4.3.4 Update the "local qpn" in the QP connection data structure sent from the remote end to the rqpn in the connection management data structure, update the "local receive buffer memory start address" in it to the raddr in the connection management data structure, and update the "local receive buffer remote access key" in it to the rkey in the connection management data structure; 4.3.5 Using qp, rqpn, raddr, and rkey from the connection management data structure as parameters, call the verbs interface to switch the QP state to a sendable and receiveable state; Fifth, the data sending module sends DDS messages using a QP connection based on the RoCE network; The data transmission module monitors the system calls of the DDS software. When the DDS software uses the socket interface to send data, it intercepts the data to be sent and uses the QP connection and connection management data structure established in step four to send the data to the receiving end via the RoCE network. The specific steps are as follows: 5.1 Monitor the system calls of the DDS software; when the DDS software calls the sendto or write interface, proceed to 5.2; otherwise, continue monitoring the system calls of the DDS software. 5.2 Intercept the sendto or write system call, obtain the data send buffer address send_addr and size send_len from the input parameters, and prevent the execution of the system call; 5.3 Data of size send_len, starting from address send_addr, is copied to the buffer of s_mr; 5.4 Construct the WR data structure s_wr for sending work requests using s_mr, rqpn, raddr, and rkey as parameters; 5.5 Using qp and s_wr as parameters, call the verbs interface to publish the work request and send the work request to the RoCE protocol stack; 5.6 Return to system call, then proceed to 5.1 to continue monitoring the DDS software's system calls; Step 6: The data receiving module receives DDS messages using a QP connection based on the RoCE network; The data receiving module monitors the system calls of the DDS software. When the DDS software uses the socket interface to receive data, it uses the QP connection and connection management data structure established in step four to complete data reception based on the RoCE network. The specific steps are as follows: 6.1 Monitor the system calls of the DDS software; when the DDS software calls recv or read, proceed to 6.2; otherwise, continue monitoring the system calls of the DDS software. 6.2 Intercept the recv or read system call, obtain the data receive buffer address recv_addr and size recv_len from the input parameters, and prevent the execution of the system call; 6.3 Copy the first recv_len bytes of data in the r_mr buffer to the memory address recv_addr; 6.4 Return to system call, go to 6.1 to continue monitoring the system calls of the DDS software.

2. The low-latency transmission method for DDS messages based on RoCE networks according to claim 1, characterized in that: Set the data transmission buffer size to 64KB.

3. The low-latency transmission method for DDS messages based on RoCE networks according to claim 1, characterized in that: Set the data receive buffer size to 64KB.

4. The low-latency transmission method for DDS messages based on RoCE networks according to claim 1, characterized in that: The specific parameters of the connection management data structure include: the socket descriptor s_id that identifies the TCP connection; the source IP address s_ip, destination IP address d_ip, source port number s_port, and destination port number d_port that identify the TCP connection; the QP connection qp that identifies the TCP connection; the CQ queue cq used by the QP connection; the QP number qpn; the data send buffer s_mr used by the QP connection; the data receive buffer r_mr; the memory address of r_mr; the descriptor tcp_cnn that indicates whether the TCP connection has been established (tcp_cnn = 1 indicates a TCP connection has been established, 0 indicates no TCP connection); the descriptor qp_cnn that indicates whether the QP connection has been established (qp_cnn = 1 indicates a QP connection has been established, 0 indicates no QP connection); the QP number rqpn of the peer connection; the address raddr of the peer connection's receive buffer; and the access key rkey of the peer connection's receive buffer.

5. The low-latency transmission method for DDS messages based on RoCE networks according to claim 1, characterized in that: The connection management data buffer size is 4K.

Images