Industrial control system data transmission method and device based on RDMA

Abstract

The invention discloses an industrial control system data transmission method and device based on RDMA, and belongs to the technical field of communication. According to the method, an RDMA transmission engine is used in the data information transmission process between a sender and a receiver, and a data buffer area is preset in the sender in advance, so that a network card can directly write in or read the buffer area specified by a user in the data transmission process, and unnecessary overhead in the data transmission process is reduced; according to the method, the memory registration area cache is introduced into the RDMA transmission engine, so that registered memory resources can be reused, the number of times and expenditure of memory registration are reduced, and the overall performance of the RDMA is improved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology and relates to an industrial control system data transmission method and device based on RDMA. Background Art

[0002] In the existing network communication field, TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two widely used transport layer protocols, which are responsible for establishing a connection between the source and the destination and transmitting data. However, these two protocols have certain performance bottlenecks in the process of data encapsulation and decapsulation.

[0003] In the data transmission process, the data generated by the source application layer first needs to be encapsulated layer by layer in the network protocol stack. Specifically, the data is first encapsulated in the transport layer (TCP or UDP), and then passes through the network layer, data link layer and physical layer in turn. Each layer will attach its own header information to the data to identify the processing information required by the layer. This process involves a large number of memory copy operations, which require frequent CPU participation, thereby occupying a large amount of CPU resources.

[0004] Similarly, during the data reception process, the data received by the destination needs to be decapsulated layer by layer starting from the physical layer. Each layer needs to parse its own header information and pass the data to the previous layer until the data reaches the application layer. This process also involves a large amount of memory copying and CPU operations, further increasing the CPU burden.

[0005] Especially for industrial control systems, this data transmission and encapsulation process may bring additional delays and overhead in industrial control systems. Especially when the network conditions are unstable or mobile devices frequently switch networks, this traditional encapsulation and decapsulation method will cause serious occupation of CPU resources, thereby affecting the performance and response speed of the entire system. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an industrial control system data transmission method and device based on RDMA to solve the problem in the prior art that CPU resources will be seriously occupied when TCP or UDP is used to transmit data in the industrial control system.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The data transmission method of industrial control system based on RDMA, the data output process includes:

[0009] The application in the sender submits a work request;

[0010] The sender converts the work request into a work queue; the work queue is registered and cached in a data cache area, which is located in the RDMA transmission engine of the sender. During the registration and caching process of the work queue, the cache is cached in the memory registration area;

[0011] The sender sends the work queue to the RDMA network card, and the RDMA network card is used to cache the data and then send it to the receiver. The receiver has established a link with the sender through a socket in the early stage;

[0012] The sender receives a confirmation character signal, which is a signal sent by the receiver after storing the data.

[0013] A further improvement of the present invention is:

[0014] Preferably, the work queue includes a source memory address, a destination memory address, a data length and a key.

[0015] Preferably, the specific process of the sender sending the work queue to the RDMA network card is that the sender network card converts the virtual address of the element in the work queue into a physical address, obtains the data to be sent from the memory based on the physical address, assembles the data to be sent into data packets and caches them in the sender network card, and sends the data cached in the sender network card to the RDMA network card.

[0016] Preferably, the receiver establishes a link with the sender via a socket in the early stage and then exchanges synchronization information; the synchronization information includes GID, QPN, VA and R_Key.

[0017] Preferably, the process of registering the area cache through the memory is: the sender calls an API or the system registers the memory area that needs to be accessed by the RDMA hardware.

[0018] The industrial control system data transmission device based on RDMA includes:

[0019] A request module, used to control the application in the sender to submit a work request;

[0020] A conversion module, used to control the sender to convert the work request into a work queue; the work queue is registered and cached in a data cache area, the data cache area is located in the RDMA transmission engine of the sender, and the work queue is cached through a memory registration area during the registration and caching process;

[0021] The sending module is used to control the sender to send the work queue to the RDMA network card, and the RDMA network card is used to cache the data and send it to the receiver. The receiver has established a link with the sender through a socket in the early stage;

[0022] The first receiving module is used to control the sender to receive a confirmation character signal, where the confirmation character signal is a signal sent by the receiver after storing the data.

[0023] The data transmission method of industrial control system based on RDMA, the data receiving process includes:

[0024] The receiving party receives a data packet from the RDMA network card; the data packet is obtained by the RDMA network card from the sending party, the sending party is provided with an RDMA transmission engine, and when the work queue in the RDMA transmission engine is registered and cached, it is cached through the memory registration area; the receiving party has established a link with the sending party through a socket in the early stage;

[0025] The receiver parses the data packet;

[0026] The receiver stores the data packet in the data buffer area, and the storage process is cached through the memory registration area;

[0027] The receiver sends a confirmation character signal to the sender.

[0028] Preferably, the specific process of the receiver parsing the data packet is that the receiver parses the virtual destination address of the received data packet, converts the parsed virtual destination address into a local physical address, and stores the parsed data in a designated area.

[0029] Preferably, the receiver stores the data packet in a designated memory area via a SIGNALED mode.

[0030] The industrial control system data transmission device based on RDMA includes:

[0031] The second receiving module is used to control the receiving party to receive a data packet from the RDMA network card; the data packet is obtained by the RDMA network card from the sending party, the sending party is provided with an RDMA transmission engine, and when the work queue in the RDMA transmission engine is registered and cached, it is cached through the memory registration area; the receiving party has established a link with the sending party through a socket in the early stage;

[0032] A parsing module, used to control the receiver to parse the data packet;

[0033] A storage module, used for controlling the receiving party to store the data packet in the data buffer area;

[0034] The feedback module is used to control the receiving party to send a confirmation character signal to the sending party.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention discloses an industrial control system data transmission method based on RDMA, which uses RDMA technology in the data information transmission process between the sender and the receiver, and realizes kernel-bypass and zero-copy data transmission methods. It enables programs in user mode to communicate directly through memory without the intervention of the operating system kernel. This direct memory access method greatly reduces unnecessary overhead in the data transmission process, such as memory copying and context switching, thereby significantly improving the efficiency and speed of data transmission.

[0037] The present invention selects a unilateral communication mode in RDMA communication. Compared with the traditional bilateral communication mode, unilateral communication only generates one message transmission and does not require the participation of the CPU. This communication mode not only reduces the burden on the CPU, but also further improves the real-time and performance of data transmission. In the power wide-area protection system, this high-performance communication mode is crucial to ensure the real-time upload of measurement data, the accurate distribution of execution messages, and the efficient information exchange between regions, thereby ensuring the safe and stable operation of the power system.

[0038] In addition, in view of the impact of memory registration on performance in RDMA operations, the present invention adopts MRRC (Memory Registration Region Cache) technology to reduce the overhead of memory registration. In RDMA operations, the network card needs to directly write or read the buffer specified by the user, so it is necessary to register these memories before each RDMA operation. However, the memory registration operation itself will bring certain overhead and may affect the performance of RDMA. By applying MRRC technology, the present invention can reuse the memory resources that have been registered, reduce the number and overhead of memory registration, and thus improve the overall performance of RDMA.

[0039] Furthermore, the present invention significantly improves the communication performance of the power wide area protection system by applying RDMA technology and unilateral communication mode, and adopting MRRC technology to reduce memory registration overhead. This not only improves the efficiency and speed of data transmission, but also reduces the burden on the CPU and enhances the real-time performance and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A data transmission diagram of an RDMA industrial control system data transmission of the present invention;

[0041] Figure 2 A data receiving diagram of an RDMA industrial control system data transmission of the present invention;

[0042] Figure 3 The present invention is a data transmission flow chart of an RDMA industrial control system. DETAILED DESCRIPTION

[0043] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0044] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0045] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] First, the abbreviations used in the present invention are explained below.

[0047] RDMA, the full name of which is Remote Direct Memory Access, transfers data directly to the computer's storage area through the network, quickly moving data from one system to the remote system's memory without causing any impact on the operating system.

[0048] Socket is an abstraction of the endpoint for two-way communication between application processes on different hosts in a network; a Socket is one end of process communication on the network, providing a mechanism for application layer processes to exchange data using network protocols; Socket is connected to the application process at the top and the network protocol stack at the bottom, and is the interface for applications to communicate through network protocols.

[0049] WRITE refers to the operation of writing data from memory to a storage medium (such as a hard disk, network, etc.).

[0050] READ refers to the operation of reading data from a storage medium (such as hard disk, memory, network, etc.)

[0051] GID refers to an identifier used to uniquely identify an entity (such as a user, device, record, etc.) in a distributed system or across multiple systems / platforms.

[0052] QPN is used to uniquely identify an RDMA queue pair (Queue Pair, referred to as QP), which includes a sending queue and a receiving queue.

[0053] ACK signal, the full name of ACK is "acknowledgement character", which is a transmission control character sent by the receiving station to the sending station, indicating that the received data has been confirmed to be received correctly.

[0054] MRRC, Memory Register Region Cache.

[0055] See also Figure 1 The first aspect of the present invention discloses a data transmission method for an industrial control system based on RDMA, which is used by a sender and is a method for outputting data during data transmission. The method comprises the following steps:

[0056] S1, the application in the sender submits a work request; it should be understood that the sender can be a client or a server; the client and the server both perform memory registration before data transmission and build their own data cache areas.

[0057] S2, the sender converts the work request into a work queue;

[0058] When a work request is converted to a work queue, a work queue element is generated, which contains the information of the request to be sent to the other end, and the work queue element is placed in the matching work queue. The work queue element contains information such as the source memory address, sender ID, destination memory address, data length, key, and send flag. The send flag indicates whether the operation is WRITE or READ.

[0059] Specifically, the source memory address refers to the address of the request information at the sender, and the destination memory address refers to the address of the request information at the receiver.

[0060] Specifically, the work queue includes a sending queue and a receiving queue, and the matching work queue is the sending queue.

[0061] The work queue is registered in the RDMA transmission engine located at the sender in the data buffer area.

[0062] The sender registers the work queues that need to be transmitted and caches them in the data buffer, which is located in the sender's RDMA transmission engine so that it can be accessed by the remote device. Then, the memory registration area cache is used to manage all registered data buffer information.

[0063] It should be understood that memory registration area caching refers to the process of registering and caching memory areas when using technologies such as RDMA. The process of memory registration area caching is as follows: the application or driver first selects a memory area that needs to be accessed by the RDMA hardware. Call the corresponding API or system to register this memory area. During this process, the operating system will lock this memory area to prevent it from being swapped to the hard disk (such as using Linux's mlock call). The size of the memory registration area cache is equal to the maximum number of memory pages allowed to be registered by the RDMA network card in the client and server. When the cache is full, a replacement algorithm is used to reselect the areas to be unregistered in batches based on size.

[0064] The RDMA hardware driver converts the virtual memory address into the physical memory address and saves this correspondence in the RDMA hardware access table.

[0065] S3, after the sender matches the work queue to the sending queue, it sends it to the RDMA network card, and the RDMA network card is used to cache the data and send it to the receiver. The receiver has established a link with the sender through a socket in the early stage.

[0066] S4, the sender receives an ACK signal, where the ACK signal is a completion signal sent by the receiver after storing the data.

[0067] In some implementation schemes of the present invention, before S1, a socket link is established between the client and the server. The link is to prepare for the formal RDMA communication. After the client and the server establish a link through the socket, they exchange synchronization information including GID (global ID), QPN (unique number of QP in a node), VA (virtual address), R_Key (key of the remote address) to complete the registration. The two ends of the successfully established RDMA will each establish a pair of queues, consisting of a sending queue and a receiving queue.

[0068] The method presets a data buffer area in the sender in advance, so that during the data transmission process, the network card can directly write or read the data buffer area specified by the user, reducing unnecessary overhead during the data transmission process.

[0069] See also Figure 2 The second aspect of the present invention discloses an industrial control system data transmission method based on RDMA, which is applied to a receiving party. The data receiving process of the receiving party includes the following steps:

[0070] S1, the receiving party receives a data packet from the RDMA network card; the data packet is obtained by the RDMA network card from the sending party;

[0071] S2, the receiver parses the data packet, which is obtained from the sender by the RDMA network card;

[0072] S3, the receiver stores the data packet in the designated memory area;

[0073] S4, the receiver sends an ACK signal to the sender.

[0074] It should be understood that the recipient can be a client or a server.

[0075] In some implementation schemes of the present invention, the specific process of S2 and S3 is: the receiver parses the virtual destination address, converts it into a local physical address, parses the data, and places the data into a designated data cache area, which also manages all registered data cache area information through a memory registration area cache.

[0076] In some implementation schemes of the present invention, a unilateral operation is used in S4. In order to let the sender know whether the operation is completed, by enabling the SIGNALED mode, after the receiver places the data into the specified memory area, the sender will receive an ACK signal and add a completion queue element to the sender's completion queue. The sender can know that the operation has been completed through the completion queue element.

[0077] A third aspect of the present invention discloses an industrial control system data transmission device based on RDMA, comprising:

[0078] A request module, used to control the application in the sender to submit a work request;

[0079] A conversion module, used to control the sender to convert the work request into a work queue; the work queue is registered and cached in a data cache area, the data cache area is located in the RDMA transmission engine of the sender, and the work queue is cached through a memory registration area during the registration and caching process;

[0080] The sending module is used to control the sender to send the work queue to the RDMA network card, and the RDMA network card is used to cache the data and send it to the receiver. The receiver has established a link with the sender through a socket in the early stage;

[0081] The first receiving module is used to control the sender to receive a confirmation character signal, where the confirmation character signal is a signal sent by the receiver after storing the data.

[0082] A fourth aspect of the present invention discloses an industrial control system data transmission device based on RDMA, comprising:

[0083] The second receiving module is used to control the receiving party to receive a data packet from the RDMA network card; the data packet is obtained by the RDMA network card from the sending party, the sending party is provided with an RDMA transmission engine, and when the work queue in the RDMA transmission engine is registered and cached, it is cached through the memory registration area; the receiving party has established a link with the sending party through a socket in the early stage;

[0084] A parsing module, used to control the receiver to parse the data packet;

[0085] A storage module is used to control the receiving party to store the data packet in the data buffer area, and the storage process is cached through the memory registration area;

[0086] The feedback module is used to control the receiving party to send an ACK signal to the sending party.

[0087] See also Figure 3 , the specific data transmission process between the sender and the receiver of the present invention is:

[0088] S1, register RDMA memory in the sender and receiver of the transmitted data

[0089] In the initial stage, both the sender and the receiver register the memory. The sender registers the data buffer to be transmitted in the RDMA transmission engine so that it can be accessed by the remote device. Then the memory registration area cache is used to manage all registered memory area information.

[0090] Among them, both the sender and the receiver's devices need to have RDMA network cards to enable RDMA transmission.

[0091] The MRRC size is equal to the maximum number of memory pages allowed to be registered by the RDMA network cards in the client and server. When the cache is full, the LRU replacement algorithm is used to reselect the areas to be deregistered in batches based on usage.

[0092] S2, establishes a link between the client and the server

[0093] The link is to prepare for the formal RDMA communication. After the client and server establish a link through the socket, they exchange synchronization information including GID (global ID), QPN (unique number of QP in a node), VA (virtual address), R_Key (key of remote address) to complete the registration. The two ends of the successfully established RDMA will each establish a pair of queues, consisting of a sending queue and a receiving queue. That is, the client and the sender each have a pair of queues.

[0094] S3: The sender (which can be a client or a server) sends a unilateral operation WRITE task in the form of a work queue element.

[0095] The sender's application submits a work request, which generates a work queue element containing information about the request to be sent to the other end, such as the operation type, control information, etc. The work queue element is placed in a matching work queue.

[0096] The work queue element includes information such as the source memory address, the destination memory address, the data length, and the key. The one-sided operation includes operations such as WRITE and READ. This description takes the WRITE operation as an example.

[0097] The work queue includes a sending queue and a receiving queue, and the matching work queue is the sending queue.

[0098] S4: The sender hardware takes the work queue element from the send queue, parses the information, and sends it to the remote RDMA network card.

[0099] The sender network card converts the virtual address in the work queue element into a physical address, then obtains the data to be sent from the memory, assembles the data packet into the network card cache, and then sends it to the remote RDMA network card.

[0100] S5, the receiver (which can be a client or a server) receives the data packet, parses the virtual destination address, converts it into a local physical address, parses the data, and places the data into a designated data buffer.

[0101] Among them, due to the unilateral operation used, in order to let the sender know whether the operation is completed, by enabling the SIGNALED mode, after the receiver places the data into the specified memory area, the sender will receive a completion signal and add a completion queue element to the sender's completion queue. The sender can know that the operation has been completed through the completion queue element.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The data transmission method of industrial control system based on RDMA is characterized by: The data output process includes: The application in the sender submits a work request; The sender converts the work request into a work queue; the work queue is registered and cached in a data cache area, which is located in the RDMA transmission engine of the sender. During the registration and caching process of the work queue, the cache is cached in the memory registration area; The sender sends the work queue to the RDMA network card, and the RDMA network card is used to cache the data and then send it to the receiver. The receiver has established a link with the sender through a socket in the early stage; The sender receives a confirmation character signal, which is a signal sent by the receiver after storing the data.

2. The RDMA-based industrial control system data transmission method according to claim 1, characterized in that: The work queue includes a source memory address, a destination memory address, a data length and a key.

3. The RDMA-based industrial control system data transmission method according to claim 1, characterized in that: The specific process of the sender sending the work queue to the RDMA network card is that the sender network card converts the virtual address of the element in the work queue into a physical address, obtains the data to be sent from the memory based on the physical address, assembles the data to be sent into a data packet and caches it in the sender network card, and sends the data cached in the sender network card to the RDMA network card.

4. The RDMA-based industrial control system data transmission method according to claim 1, characterized in that: After the receiver establishes a link with the sender through the socket in the early stage, they exchange synchronization information; the synchronization information includes GID, QPN, VA and R_Key.

5. The RDMA-based industrial control system data transmission method according to claim 1, characterized in that: The process of registering the area cache through the memory is as follows: the sender calls an API or the system registers the memory area that needs to be accessed by the RDMA hardware; The data size of the memory registration area cache is the maximum number of memory pages allowed to be registered by the RDMA network card.

6. An industrial control system data transmission device based on RDMA, characterized in that: include: A request module, used to control the application in the sender to submit a work request; A conversion module, used to control the sender to convert the work request into a work queue; The work queue is registered and cached in a data cache area, which is located in the RDMA transmission engine of the sender. During the registration and caching process of the work queue, the cache is cached through the memory registration area; The sending module is used to control the sender to send the work queue to the RDMA network card, and the RDMA network card is used to cache the data and send it to the receiver. The receiver has established a link with the sender through a socket in the early stage; The first receiving module is used to control the sender to receive a confirmation character signal, where the confirmation character signal is a signal sent by the receiver after storing the data.

7. The data transmission method of industrial control system based on RDMA is characterized in that: The data receiving process includes: The receiving party receives a data packet from the RDMA network card; the data packet is obtained by the RDMA network card from the sending party, the sending party is provided with an RDMA transmission engine, and when the work queue in the RDMA transmission engine is registered and cached, it is cached through the memory registration area; the receiving party has established a link with the sending party through a socket in the early stage; The receiver parses the data packet; The receiver stores the data packet in the data buffer area, and the storage process is cached through the memory registration area; The receiver sends a confirmation character signal to the sender.

8. The RDMA-based industrial control system data transmission method according to claim 5, characterized in that: The specific process of the receiver parsing the data packet is that the receiver parses the virtual destination address of the received data packet, converts the parsed virtual destination address into a local physical address, and stores the parsed data in a designated area.

9. The RDMA-based industrial control system data transmission method according to claim 5, characterized in that: The receiver stores the data packet in a designated memory area through the SIGNALED mode.

10. An industrial control system data transmission device based on RDMA, characterized in that: include: The second receiving module is used to control the receiving party to receive data packets from the RDMA network card; The data packet is obtained by the RDMA network card from the sender, the sender is provided with an RDMA transmission engine, and when the work queue in the RDMA transmission engine is registered and cached, it is cached through the memory registration area; The receiver has previously established a connection with the sender via a socket; A parsing module, used to control the receiver to parse the data packet; A storage module, used for controlling the receiving party to store the data packet in the data buffer area; The feedback module is used to control the receiving party to send a confirmation character signal to the sending party.

Images