Data reading and writing method, device, equipment, and medium

Abstract

The present application discloses a data reading and writing method, apparatus, device, and medium, which includes: using a bilateral interface RDMA recv to receive a data reading and writing request sent by a storage client on a second node in the distributed storage system; obtaining a unilateral memory address from the data reading and writing request; performing an RDMA unilateral data reading and writing operation on the memory corresponding to the unilateral memory address in response to the data reading and writing request; using a bilateral interface RDMA send to send a reply message corresponding to the data reading and writing request to the storage client on the second node, so that the storage client on the second node receives the reply message using the bilateral interface RDMA recv, wherein the reply message is used to indicate that the operation corresponding to the data reading and writing request has been completed. This is compatible with existing network communication architectures, reduces data reading and writing latency, improves IOPS, and saves communication overhead.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data reading and writing method, device, equipment, and medium. Background Art

[0002] With the development of artificial intelligence and 5G technology, customers' businesses and applications are constantly enriched, and the amount of data that needs to be stored and transmitted is constantly increasing. However, distributed storage systems use traditional TCP / IP (Transmission Control Protocol / Internet Protocol) protocols for cross-node data communication. The cross-node network latency is high, which increases user IO (Input / Output) latency and reduces system IOPS (Input / Output Operations Per Second). Network performance has become a bottleneck restricting hyper-converged read and write performance.

[0003] To solve the above problems, RDMA (Remote Direct Memmory Access) technology and network cards were developed. These allow data to be moved directly between application memories on different servers, bypassing the kernel protocol stack (Kernel Bypass) without the need for CPU (central processing unit) intervention, thereby improving network performance.

[0004] However, the RDMA technologies currently proposed only use bilateral primitives for data transmission, or only use unilateral primitives for data transmission. For example, Baidu RPC (Brpc) is Baidu's RPC (Remote Procedure Call) framework. By establishing a Socket-like RDMA communication library, the client first establishes a TCP (Transmission Control Protocol) connection with the server, and then performs a handshake to establish an RDMA connection through RDMA cm (RDMA Connection Management). Data transmission uses RDMA's bilateral interface RDMA send / RDMA recv. Brpc only uses RDMA's bilateral interface, but bilateral operations have high latency and low IOPS. Of course, there are also RDMA technologies that only use unilateral operations. Since unilateral operations require advance negotiation of the address, offset, and key of the memory operation, and different applications will directly use the memory requested by the business to avoid copying, the memory address is constantly changing, so each communication requires multiple interactions, increasing communication overhead. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a data reading and writing method, device, equipment, and medium that are compatible with existing network communication architectures, reduce data reading and writing latency, increase IOPS, and save communication overhead. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a data reading and writing method, which is applied to a storage service client on a first node in a distributed storage system, comprising:

[0007] Receiving, using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, wherein the data read / write request is sent by the storage client on the second node using the bilateral interface RDMA send, and an RDMA connection established based on the RoCE protocol exists between the storage service client on the first node and the storage client on the second node;

[0008] Obtaining a unilateral memory address from the data read / write request, wherein the unilateral memory address is an address in a first unilateral memory on the second node for storing read / write data transmitted between the first node and the second node through the RDMA connection;

[0009] Performing an RDMA unilateral data read and write operation on the memory corresponding to the unilateral memory address to respond to the data read and write request;

[0010] The bilateral interface RDMA send is used to send reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed.

[0011] Optionally, before receiving the data read / write request sent by the storage client on the second node in the distributed storage system by using the bilateral interface RDMA recv, the method further includes:

[0012] When the storage service client on the first node is initialized, applying for and registering a first bilateral memory and a second bilateral memory, and pre-registering a second unilateral memory, wherein the first bilateral memory is a memory in the first node used to store data received through the bilateral interface RDMA recv, the second bilateral memory is a memory in the first node used to store data sent through the bilateral interface RDMA send, and the second unilateral memory is a memory in the first node used to store data to be written transmitted between the first node and the second node through the RDMA connection;

[0013] When the storage service client on the first node receives the connection establishment request sent by the storage client on the second node, it creates connection resources corresponding to the RDMA connection and establishes the RDMA connection.

[0014] Optionally, the receiving, by using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system includes:

[0015] Retrieving a target receiving bilateral memory address from a locally stored receiving address set, wherein the address in the receiving address set is the address in the first bilateral memory;

[0016] A data read / write request sent by a storage client on a second node in the distributed storage system is received using a bilateral interface RDMA recv, and information included in the data read / write request is stored in the target receiving bilateral memory address.

[0017] Optionally, the sending reply information corresponding to the data read and write request to the storage client on the second node by using the bilateral interface RDMA send includes:

[0018] Retrieving a target sending bilateral memory address from a locally stored sending address set, wherein the address in the sending address set is the address in the second bilateral memory;

[0019] Writing the reply information corresponding to the data read / write request to the target sending bilateral memory address and submitting the reply information to the reply information queue;

[0020] Check whether the reply information meets the sending condition, and if the reply information meets the sending condition, send the reply information to the storage client on the second node by using the bilateral interface RDMA send.

[0021] Optionally, performing an RDMA unilateral data read and write operation on the memory corresponding to the unilateral memory address includes:

[0022] When the data read / write request is a data write request, an RDMA unilateral data read operation is performed on the memory corresponding to the unilateral memory address to obtain the data to be written, and the data to be written is written into the second unilateral memory.

[0023] Optionally, performing an RDMA unilateral data read and write operation on the memory corresponding to the unilateral memory address includes:

[0024] When the data read / write request is a data read request, the to-be-read data corresponding to the data read request is read from the storage device of the first node, and the to-be-read data is written to the memory corresponding to the unilateral memory address.

[0025] Optionally, the method further includes:

[0026] When the RDMA connection is destroyed, the first bilateral memory, the second bilateral memory, the second unilateral memory, and the connection resources are released.

[0027] In a second aspect, the present application discloses a data reading and writing device, which is applied to a storage service client on a first node in a distributed storage system, comprising:

[0028] a request acquisition module, configured to receive, using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, wherein the data read / write request is sent by the storage client on the second node using the bilateral interface RDMA send, and an RDMA connection established based on the RoCE protocol exists between the storage service client on the first node and the storage client on the second node;

[0029] an information extraction module, configured to obtain a unilateral memory address from the data read / write request, wherein the unilateral memory address is an address in a first unilateral memory on the second node for storing read / write data transmitted between the first node and the second node via the RDMA connection;

[0030] A read / write module, configured to perform RDMA unilateral data read / write operations on the memory corresponding to the unilateral memory address to respond to the data read / write request;

[0031] a reply information sending module, configured to use the bilateral interface RDMA send to send reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed.

[0032] In a third aspect, the present application discloses an electronic device, comprising:

[0033] memory and processor;

[0034] Wherein, the memory is used to store computer programs;

[0035] The processor is used to execute the computer program to implement the aforementioned disclosed data reading and writing method.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed data reading and writing method when executed by a processor.

[0037] It can be seen that the present application is applied to the storage service client on the first node in the distributed storage system, and first uses the bilateral interface RDMA recv to receive the data read and write request sent by the storage client on the second node in the distributed storage system, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send, and there is an RDMA connection established based on the RoCE protocol between the storage service client on the first node and the storage client on the second node. Then, a unilateral memory address is obtained from the data read and write request, wherein the unilateral memory address is an address in the first unilateral memory on the second node for storing the read and write data transmitted between the first node and the second node through the RDMA connection. Then, an RDMA unilateral data read and write operation is performed on the memory corresponding to the unilateral memory address to respond to the data read and write request. And the bilateral interface RDMA send is used to send the reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed. It can be seen that in this application, the storage service end on the first node uses the bilateral interface RDMArecv to receive the data read and write request sent by the storage client on the second node using the bilateral interface RDMA send, and then the storage service end on the first node performs a unilateral data read and write operation on the memory corresponding to the unilateral memory address in the data read and write request to respond to the data read and write request. Then the storage service end on the first node also uses the bilateral interface RDMAsend to send the reply information of the request completion to the storage client on the second node. In this way, there is an RDMA connection established based on the RoCE protocol between the storage service end on the first node and the storage client on the second node. In the process of implementing RDMA between the first node and the second node, the bilateral interfaces RDMA send and RDMA recv are used to send and receive data read and write requests and reply information. Since the RDMA connection is established based on the RoCE protocol, and the sending and receiving of data read and write requests and reply information use the bilateral interfaces RDMA send and RDMA recv, it is compatible with the existing network communication structure. In addition, the two nodes in the distributed storage system transmit data based on RDMA technology, bypassing the kernel protocol stack, reducing system read and write latency and reducing CPU usage.Furthermore, the data read / write request already includes the unilateral memory address. Therefore, only the unilateral data read / write operation needs to be performed on the memory corresponding to the unilateral memory address. Data to be written to the first node can be read from the unilateral memory address, or data read from the first node can be written to the memory corresponding to the unilateral memory address. This allows RDMA between nodes to combine unilateral and bilateral operations, reducing data read / write latency and increasing IOPS. Furthermore, since the unilateral memory address corresponding to the unilateral data read / write operation is sent to the storage service client on the first node in the data read / write request, the problem of multiple communications between the storage service client on the first node and the storage client on the second node to obtain the operation address can be avoided, saving communication overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0039] Figure 1 A flow chart of a data reading and writing method disclosed in this application;

[0040] Figure 2 A flow chart of a specific data reading and writing method disclosed in this application;

[0041] Figure 3 A schematic diagram of a specific distributed storage system disclosed in this application;

[0042] Figure 4 A specific data writing flow chart disclosed in this application;

[0043] Figure 5 A specific data reading flow chart disclosed in this application;

[0044] Figure 6 A flow chart of a data reading and writing process disclosed in this application;

[0045] Figure 7 This is a structural diagram of a data reading and writing device disclosed in this application;

[0046] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] See also Figure 1 As shown, an embodiment of the present application discloses a data reading and writing method, which is applied to a storage service client on a first node in a distributed storage system. The method includes:

[0049] Step S11: Use the bilateral interface RDMA recv to receive a data read and write request sent by the storage client on the second node in the distributed storage system, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send, and there is an RDMA connection established based on the RoCE protocol between the storage service client on the first node and the storage client on the second node.

[0050] The distributed storage system includes a first node and a second node, wherein the first node can be any node in the distributed storage system, and the second node can be any node in the distributed storage system other than the first node. The first node and the second node each include their own storage service client, storage client, and storage device, and the storage device includes but is not limited to a solid-state drive. That is, the first node includes a storage client and a storage service client, and the second node also includes a storage client and a storage service client. RDMA connections established based on the RoCE (RDMA over Convergent Ethernet) protocol exist between the storage client on the first node and the storage service client on each node in the distributed storage system other than the first node. RDMA connections established based on the RoCE protocol also exist between the storage client on the second node and the storage service client on each node in the distributed storage system other than the second node. The storage client needs to receive data read and write requests issued by the application on its own node in order to call the storage service client on its own node or the storage service client on other nodes to complete the corresponding operations of the data read and write requests, and the storage service client needs to manage the mounted storage devices.

[0051] The storage service end on the first node needs to use the bilateral interface RDMA recv to receive the data read and write request sent by the storage client on the second node, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send.

[0052] That is, when the second node needs to perform an RDMA operation on the first node, the storage client on the second node will use the bilateral interface RDMA send to send a data read request to the storage service end on the first node. Correspondingly, the storage service end on the first node needs to use the bilateral interface RDMA recv to receive the data read and write request, wherein the data read and write request includes a unilateral memory address, offset and key, and the unilateral memory address is the address in the first unilateral memory on the second node for storing the read and write data transmitted between the first node and the second node via the RDMA connection. That is, the first unilateral memory is used to store the data to be written to the first node that needs to be transmitted via the RDMA connection between the first node and the second node, and to store the data to be read from the first node and returned to the second node that needs to be transmitted via the RDMA connection between the first node and the second node.

[0053] Step S12: Obtain a unilateral memory address from the data read and write request, wherein the unilateral memory address is an address in the first unilateral memory on the second node, used to store the read and write data transmitted between the first node and the second node through the RDMA connection.

[0054] It is understandable that after receiving the data read / write request, the storage service client on the first node also needs to obtain the unilateral memory address from the data read / write request. In addition to the unilateral memory address, it also needs to obtain the offset and the key. After obtaining the unilateral memory address, the storage service client on the first node knows where to read the data to be written and write it to the first node, and where to write the data to be read read from the storage device by the storage service client on the first node to the second node.

[0055] Step S13: performing an RDMA unilateral data read / write operation on the memory corresponding to the unilateral memory address to respond to the data read / write request.

[0056] It is understandable that after obtaining the unilateral memory address, an RDMA unilateral data read / write operation can be performed on the memory corresponding to the unilateral memory address to respond to the data read / write request. The RDMA unilateral data read / write operation is also the RDMA READ / Write operation in RDMA technology. Since the RDMA READ / Write operation in RDMA technology is based on the unilateral primitive operation, the RDMA READ / Write operation in RDMA technology is referred to as the RDMA unilateral data read / write operation here.

[0057] Specifically, when the data read / write request is a data write request, an RDMA unilateral data read operation is performed on the memory corresponding to the unilateral memory address to obtain the data to be written, and the data to be written is written to the second unilateral memory on the first node. The second unilateral memory is a memory in the first node used to store the data to be written transmitted between the first node and the second node via the RDMA connection. After the data to be written is written to the first unilateral memory, the storage service client on the first node can store the data to be written in the storage device in the first node according to its own data disk placement mechanism.

[0058] When the data read / write request is a data read request, the to-be-read data corresponding to the data read request is read from the storage device of the first node, and the to-be-read data is written to the memory corresponding to the unilateral memory address.

[0059] Step S14: Use the bilateral interface RDMA send to send a reply message corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply message using the bilateral interface RDMA recv, wherein the reply message is used to indicate that the operation corresponding to the data read and write request has been completed.

[0060] After responding to the data read / write request, the storage service on the first node also needs to notify the storage client on the second node that the operation corresponding to the data read / write request has been completed. Specifically, the storage service on the first node uses the bilateral interface RDMA send to send the reply information corresponding to the data read / write request to the storage client on the second node, and the storage client on the second node then uses the bilateral interface RDMA recv to receive the reply information, wherein the reply information is used to indicate that the operation corresponding to the data read / write request has been completed.

[0061] It can be seen that the present application is applied to the storage service client on the first node in the distributed storage system, and first uses the bilateral interface RDMA recv to receive the data read and write request sent by the storage client on the second node in the distributed storage system, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send, and there is an RDMA connection established based on the RoCE protocol between the storage service client on the first node and the storage client on the second node. Then, a unilateral memory address is obtained from the data read and write request, wherein the unilateral memory address is an address in the first unilateral memory on the second node for storing the read and write data transmitted between the first node and the second node through the RDMA connection. Then, an RDMA unilateral data read and write operation is performed on the memory corresponding to the unilateral memory address to respond to the data read and write request. And the bilateral interface RDMA send is used to send the reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed. It can be seen that in this application, the storage service end on the first node uses the bilateral interface RDMArecv to receive the data read and write request sent by the storage client on the second node using the bilateral interface RDMA send, and then the storage service end on the first node performs a unilateral data read and write operation on the memory corresponding to the unilateral memory address in the data read and write request to respond to the data read and write request. Then the storage service end on the first node also uses the bilateral interface RDMAsend to send the reply information of the request completion to the storage client on the second node. In this way, there is an RDMA connection established based on the RoCE protocol between the storage service end on the first node and the storage client on the second node. In the process of implementing RDMA between the first node and the second node, the bilateral interfaces RDMA send and RDMA recv are used to send and receive data read and write requests and reply information. Since the RDMA connection is established based on the RoCE protocol, and the sending and receiving of data read and write requests and reply information use the bilateral interfaces RDMA send and RDMA recv, it is compatible with the existing network communication structure. In addition, the two nodes in the distributed storage system transmit data based on RDMA technology, bypassing the kernel protocol stack, reducing system read and write latency and reducing CPU usage.Furthermore, the data read / write request already includes the unilateral memory address. Therefore, only the unilateral data read / write operation needs to be performed on the memory corresponding to the unilateral memory address. Data to be written to the first node can be read from the unilateral memory address, or data read from the first node can be written to the memory corresponding to the unilateral memory address. This allows RDMA between nodes to combine unilateral and bilateral operations, reducing data read / write latency and increasing IOPS. Furthermore, since the unilateral memory address corresponding to the unilateral data read / write operation is sent to the storage service client on the first node in the data read / write request, the problem of multiple communications between the storage service client on the first node and the storage client on the second node to obtain the operation address can be avoided, saving communication overhead.

[0062] See also Figure 2 As shown, the embodiment of the present application discloses a specific data reading and writing method, which is applied to a storage service client on a first node in a distributed storage system. The method includes:

[0063] Step S21: When the storage service client on the first node is initialized, a first bilateral memory and a second bilateral memory are applied for and registered, and a second unilateral memory is pre-registered, wherein the first bilateral memory is a memory in the first node used to store data received through the bilateral interface RDMA recv, the second bilateral memory is a memory in the first node used to store data sent through the bilateral interface RDMA send, and the second unilateral memory is a memory in the first node used to store data to be written transmitted between the first node and the second node through the RDMA connection.

[0064] In practical applications, the storage service and storage client on each node in the distributed storage system need to be initialized first, so that the storage client on one node can perform RDMA operations with the storage service on other nodes except its own node.

[0065] Specifically, when the storage service client on the first node is initialized, it applies for and registers the first bilateral memory and the second bilateral memory, and pre-registers the second unilateral memory, wherein the first bilateral memory is the memory in the first node used to store data received through the bilateral interface RDMA recv, the second bilateral memory is the memory in the first node used to store data sent through the bilateral interface RDMA send, and the second unilateral memory is the memory in the first node used to store the data to be written transmitted between the first node and the second node through the RDMA connection. In addition, the storage service client on the first node also needs to create an RDMA cm event channel, register an RDMA cm event processing function, and create a cm_id (Identity) of the RDMA_PS_TCP type, and call the RDMA_listen interface in the RDMA dynamic library on the first node to start the listening service to listen to events sent by storage clients on nodes other than the first node in the distributed storage system.

[0066] Accordingly, when the storage client on the second node is initialized, it applies for and registers the third bilateral memory and the fourth bilateral memory, and pre-registers the first unilateral memory, wherein the third bilateral memory is the memory in the second node used to store data received through the bilateral interface RDMA recv, the fourth bilateral memory is the memory in the second node used to store data sent through the bilateral interface RDMA send, and the first unilateral memory is the memory in the second node used to store data to be written transmitted between the first and second nodes through the RDMA connection or data to be read returned from the first node. In addition, the storage client on the second node also needs to create an RDMA cm event channel and register an event handling function for the RDMA cm event.

[0067] Step S22: When the storage service client on the first node receives the connection establishment request sent by the storage client on the second node, it creates a connection resource corresponding to the RDMA connection and establishes the RDMA connection.

[0068] It is understandable that an RDMA connection between the storage service client on the first node and the storage client on the second node needs to be established based on the RoCE protocol. Specifically, when the storage service client on the first node receives a connection establishment request sent by the storage client on the second node, it creates a connection resource corresponding to the RDMA connection and establishes the RDMA connection.

[0069] In actual applications, when the storage service client on the first node receives a connection establishment request sent by the storage client on the second node, the entire process of creating the connection resources corresponding to the RDMA connection and establishing the RDMA connection includes: the storage client on the second node calls the RDMA_resolve_addr interface to map the IP address (Internet Protocol) and port of the storage service client on the first node to an available RDMA address, and generates an RDMA CM event of the RDMA_CM_EVENT_ADDR_RESOLVED type. The storage service client on the first node then processes the RDMA_CM_EVENT_ADDR_RESOLVED event, obtains the mapped available RDMA address, and calls the RDMA_resolve_route interface to obtain the RDMA routing information required for the connection. If successful, an RDMA CM event RDMA_CM_EVENT_ROUTE_RESOLVED is generated.

[0070] The storage service client on the first node receives the RDMA_CM_EVENT_ROUTE_RESOLVED event and, based on the maximum supported connection depth, creates the connection resources required for the RDMA connection: a QP (Queue Pair), a CQ (Completion Queue), an SQ (Send Queue), a receiving address set, a sending address set, and a comp_channel (the completion queue for connection creation). It then calls the ibv_post_recv interface to push the receiving address set to the receiving queue and registers the connection event handler based on the comp_channel event Fd (file descriptor). The storage client on the second node calls the RDMA_connect interface to initiate a connection establishment request to the storage service client on the first node. Upon receiving the connection establishment request, the storage service client on the first node generates an RDMA CM event of the RDMA_CM_EVENT_CONNECT_REQUEST type, creates a new child_cm_id, and creates a new RDMA connection. The required connection resources (QP, CQ, SQ, receiving address set, sending address set, and comp_channel) are initialized according to the maximum depth. The ibv_post_recv interface is called to push the receiving address set to the receiving queue. The connection event handler is registered based on the comp_channel event Fd. RDMA_accept is called to accept the connection establishment request and establish the RDMA connection. Both the storage service client on the first node and the storage client on the second node receive an RDMA CM event of the RDMA_CM_EVENT_ESTABLISHED type, indicating a successful connection establishment. The addresses in the receiving address set are those in the first bilateral memory, and the addresses in the sending address set are those in the second bilateral memory. The maximum depth is the maximum number of data packets that can be sent continuously by the RDMA without waiting for a reply. Once the maximum depth is reached, a reply must be received before the next packet can be sent.

[0071] The storage client on the second node is the same as the storage service client on the first node and also needs to create connection resources corresponding to the RDMA connection. After the storage client on the second node successfully establishes the RDMA connection, the connection status is set to 1.

[0072] Step S23: Use the bilateral interface RDMA recv to receive a data read and write request sent by the storage client on the second node in the distributed storage system, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send, and there is an RDMA connection established based on the RoCE protocol between the storage service client on the first node and the storage client on the second node.

[0073] When the second node needs to perform an RDMA operation on the first node, the storage client on the second node will use the bilateral interface RDMA send to send a data read request to the storage service client on the first node. Correspondingly, the storage client on the first node will need to use the bilateral interface RDMA recv to receive the data read and write request. Specifically, the target receiving bilateral memory address is taken from the locally stored receiving address set; the bilateral interface RDMA recv is used to receive the data read and write request sent by the storage client on the second node in the distributed storage system, and the information included in the data read and write request is stored under the target receiving bilateral memory address.

[0074] Step S24: Obtain a unilateral memory address from the data read and write request, wherein the unilateral memory address is an address in the first unilateral memory on the second node, used to store the read and write data transmitted between the first node and the second node through the RDMA connection.

[0075] Step S25: performing an RDMA unilateral data read / write operation on the memory corresponding to the unilateral memory address to respond to the data read / write request.

[0076] The specific implementation process of step S24 and step S25 can refer to the specific content disclosed in the above embodiments, and will not be repeated here.

[0077] Step S26: Use the bilateral interface RDMA send to send a reply message corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply message using the bilateral interface RDMA recv, wherein the reply message is used to indicate that the operation corresponding to the data read and write request has been completed.

[0078] After the storage service client on the first node responds to the data read and write request, the storage service client on the first node also needs to use the bilateral interface RDMA send to send reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node can receive the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed.

[0079] Specifically, the storage service client on the first node takes out the target sending bilateral memory address from the locally stored sending address set, wherein the address in the sending address set is the address in the second bilateral memory; writes the reply information corresponding to the data read and write request to the target sending bilateral memory address, and submits the reply information to the reply information queue; checks whether the reply information meets the sending conditions, and when the reply information meets the sending conditions, uses the bilateral interface RDMA send to send the reply information to the storage client on the second node. Checking whether the reply information meets the sending conditions is specifically to check whether the remaining depth of the first node is 0, that is, the difference between the maximum depth and the number of data packets that the storage service client of the first node has sent to the storage client of the second node but has not received reply information. If not, and all the information queued before the reply information in the reply information queue is sent, then the reply information meets the sending conditions.

[0080] The specific process of the storage client on the second node sending the data read and write request is the same as the process of the storage service client on the first node sending the reply information.

[0081] Step S27: when the RDMA connection is terminated, releasing the first bilateral memory, the second bilateral memory, the second unilateral memory and the connection resources.

[0082] If the RDMA connection experiences an abnormal state in the storage service client on the first node, an RDMA_cm exception event type will be received. Exception handling will be performed based on the specific abnormal event to ensure the robustness of the connection. If the RDMA connection is terminated, the first bilateral memory, the second bilateral memory, the second unilateral memory, and the connection resources must be released. If the RDMA connection is terminated, the storage client on the second node must set the connection state to 0 and then release the connection resources in the storage client on the second node.

[0083] See also Figure 3The figure shows a schematic diagram of a distributed storage system. The entire system consists of a virtual machine / client module and a distributed storage system. The virtual machine / client module is also an upper-layer application. The virtual machine / client module sends data read and write requests to the storage system, wherein the storage system includes a storage client, a storage service client, and a storage device on each node. The storage client on each node is connected to the storage service client of other nodes through an RDMA connection established based on RoCE, and data is transmitted across nodes using the RoCE protocol. Each storage service client and storage client includes a RoCE module and an RPC module. The RPC module calls the RoCE module at the transport layer to perform data read and write requests and data transmission to be read and written, and finally returns the data to the virtual machine / client module after obtaining the data from other nodes. Among them, the aforementioned steps S11 to S14, and steps S21 to S27 can be specifically completed by the RoCE module in the storage service client on the first node.

[0084] See also Figure 4The figure shows a data write flow chart. The RoCE module in the storage client on the second node initiates a write request write_request (i.e., the aforementioned data write request) and adds the request to the tail of the request send queue corresponding to the RDMA connection between the storage client on the second node and the storage service client on the first node. The RoCE module in the storage client on the second node checks the remaining depth of this connection to determine whether the sending conditions are met; if so, the write request write_request is sent using the bilateral interface RDMA send, and the remaining depth quota is reduced by 1, otherwise continue waiting. The RoCE module in the storage service client on the first node receives the write request write_request using the bilateral interface RDMA recv, and calls the ibv_poll_cq interface to obtain the write request write_request; according to the data memory address (i.e., the aforementioned unilateral memory address), offset, and key (i.e., the aforementioned key) in the write request write_request, the RoCE module performs an RDMA READ operation (i.e., the aforementioned RDMA unilateral data read operation), obtains the data to be written from the second node and places it in the unilateral memory of the first node, and the RoCE module in the storage service client on the first node notifies the RPC module in the storage service client on the first node of the write request write_request and the data to be written; the RoCE module in the storage service client on the first node submits a reply message reply to the tail of the reply message queue corresponding to this connection, checks the queue depth (i.e., the aforementioned remaining depth), and determines whether the reply message reply can be sent; if so, uses the bilateral interface RDMA Send sends a reply message, and the queue depth quota is reduced by 1; otherwise, continue waiting; the RoCE module in the storage client on the second node calls the ibv_poll_cq interface to receive the reply message, obtains the reply message and returns it to its own RPC module, and its own queue depth quota is increased by 1 to restore the queue depth. The RoCE module in the storage service client on the first node receives the reply message reply completion prompt, and its own queue depth quota is increased by 1.

[0085] See also Figure 5The figure shows a data read flow chart. The RoCE module in the storage client on the second node initiates a read request read_request (that is, the aforementioned data read request), and adds the request to the tail of the request sending queue corresponding to the RDMA connection between the storage client on the second node and the storage service client on the first node. The RoCE module in the storage client on the second node determines whether the sending conditions are met by checking the remaining depth of this connection; if so, the read request read_request is sent using the bilateral interface RDMA send, and the remaining depth quota is reduced by 1; otherwise, continue to wait. The storage service client on the first node receives the read request read_request using the bilateral interface RDMA recv, and the RoCE module in the storage service client on the first node calls the ibv_poll_cq interface to obtain the read request read_request; according to the data memory address (that is, the aforementioned unilateral memory address), offset and key (that is, the aforementioned key) in the read request read_request, the RDMA is executed. The write operation (that is, the aforementioned RDMA unilateral data write operation) writes the to-be-read data read from the first node into the memory corresponding to the data memory address, and the RoCE module in the storage service client on the first node notifies the RPC module in the storage service client on the first node; the RoCE module in the storage service client on the first node submits the reply message reply to the tail of the reply message queue corresponding to this connection, checks the queue depth (that is, the aforementioned remaining depth), and determines whether the reply message reply can be sent; if so, the bilateral interface RDMAsend is used to send the reply message reply, and the queue depth quota is reduced by 1; otherwise, continue to wait; the RoCE module in the storage client on the second node calls the ibv_poll_cq interface to receive the reply message reply, obtains the reply message reply and returns it to its own RPC module, and its own queue depth quota is increased by 1 to restore the queue depth. The RoCE module in the storage service client on the first node receives the reply message reply completion prompt, and its own queue depth quota is increased by 1.

[0086] See also Figure 6As shown, it is a flow chart of the data reading and writing processing process. After the RoCE module in the storage client on the second node obtains the RPC request sent by the RPC module, it needs to establish a connection, transmit data, and destroy the connection. The connection establishment in the RoCE module in the storage client on the second node includes RDMA resource initialization and initiating a connection RDMA_connect to the storage service client on the first node, and establishing an RDMA connection with the storage service client on the first node through the RDMA cm event mechanism. The connection establishment of the RoCE module in the storage service client on the first node includes RDMA resource initialization, listening RDMA listen, and accepting the connection RDMA accetp. In the process of the storage client on the second node submitting a data read and write request to the storage service client on the first node, the request and reply are transmitted bilaterally, that is, data read and write requests are sent and reply information is received through RDMA send / RDMA recv, and data packets are transmitted unilaterally, that is, the data to be written is obtained through a unilateral data read operation (RDMA read) and written into the second unilateral memory of the first node, and the data to be read read from the first node is written into the first unilateral memory of the second node through a unilateral data write operation (RDMA write). Connection destruction mainly uses the RDMA cm event mechanism to handle abnormal connection states and release connection resources when the connection is destroyed.

[0087] See also Figure 7 As shown, an embodiment of the present application discloses a data reading and writing device, which is applied to a storage service client on a first node in a distributed storage system, including:

[0088] a request acquisition module 11, configured to receive, using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, wherein the data read / write request is sent by the storage client on the second node using the bilateral interface RDMA send, and an RDMA connection based on the RoCE protocol exists between the storage service client on the first node and the storage client on the second node;

[0089] an information extraction module 12, configured to obtain a unilateral memory address from the data read / write request, wherein the unilateral memory address is an address in a first unilateral memory on the second node for storing read / write data transmitted between the first node and the second node via the RDMA connection;

[0090] The read / write module 13 is configured to perform RDMA unilateral data read / write operations on the memory corresponding to the unilateral memory address to respond to the data read / write request;

[0091] The reply information sending module 14 is used to send the reply information corresponding to the data read and write request to the storage client on the second node using the bilateral interface RDMA send, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed.

[0092] It can be seen that the present application is applied to the storage service client on the first node in the distributed storage system, and first uses the bilateral interface RDMA recv to receive the data read and write request sent by the storage client on the second node in the distributed storage system, wherein the data read and write request is sent by the storage client on the second node using the bilateral interface RDMA send, and there is an RDMA connection established based on the RoCE protocol between the storage service client on the first node and the storage client on the second node. Then, a unilateral memory address is obtained from the data read and write request, wherein the unilateral memory address is an address in the first unilateral memory on the second node for storing the read and write data transmitted between the first node and the second node through the RDMA connection. Then, an RDMA unilateral data read and write operation is performed on the memory corresponding to the unilateral memory address to respond to the data read and write request. And the bilateral interface RDMA send is used to send the reply information corresponding to the data read and write request to the storage client on the second node, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read and write request has been completed. It can be seen that in this application, the storage service end on the first node uses the bilateral interface RDMArecv to receive the data read and write request sent by the storage client on the second node using the bilateral interface RDMA send, and then the storage service end on the first node performs a unilateral data read and write operation on the memory corresponding to the unilateral memory address in the data read and write request to respond to the data read and write request. Then the storage service end on the first node also uses the bilateral interface RDMAsend to send the reply information of the request completion to the storage client on the second node. In this way, there is an RDMA connection established based on the RoCE protocol between the storage service end on the first node and the storage client on the second node. In the process of implementing RDMA between the first node and the second node, the bilateral interfaces RDMA send and RDMA recv are used to send and receive data read and write requests and reply information. Since the RDMA connection is established based on the RoCE protocol, and the sending and receiving of data read and write requests and reply information use the bilateral interfaces RDMA send and RDMA recv, it is compatible with the existing network communication structure. In addition, the two nodes in the distributed storage system transmit data based on RDMA technology, bypassing the kernel protocol stack, reducing system read and write latency and reducing CPU usage.Furthermore, the data read / write request already includes the unilateral memory address. Therefore, only the unilateral data read / write operation needs to be performed on the memory corresponding to the unilateral memory address. Data to be written to the first node can be read from the unilateral memory address, or data read from the first node can be written to the memory corresponding to the unilateral memory address. This allows RDMA between nodes to combine unilateral and bilateral operations, reducing data read / write latency and increasing IOPS. Furthermore, since the unilateral memory address corresponding to the unilateral data read / write operation is sent to the storage service client on the first node in the data read / write request, the problem of multiple communications between the storage service client on the first node and the storage client on the second node to obtain the operation address can be avoided, saving communication overhead.

[0093] In some specific implementations, the data reading and writing device further includes:

[0094] An initialization module is used to apply for and register a first bilateral memory and a second bilateral memory, and pre-register a second unilateral memory when the storage service client on the first node is initialized, wherein the first bilateral memory is a memory in the first node used to store data received through the bilateral interface RDMA recv, the second bilateral memory is a memory in the first node used to store data sent through the bilateral interface RDMA send, and the second unilateral memory is a memory in the first node used to store data to be written transmitted between the first node and the second node through the RDMA connection; when the storage service client on the first node receives a connection establishment request sent by the storage client on the second node, it creates a connection resource corresponding to the RDMA connection and establishes the RDMA connection.

[0095] In some specific implementations, the request acquisition module 11 is used to:

[0096] Retrieving a target receiving bilateral memory address from a locally stored receiving address set, wherein the address in the receiving address set is the address in the first bilateral memory;

[0097] A data read / write request sent by a storage client on a second node in the distributed storage system is received using a bilateral interface RDMA recv, and information included in the data read / write request is stored in the target receiving bilateral memory address.

[0098] In some specific implementations, the reply information sending module 14 is configured to:

[0099] Retrieving a target sending bilateral memory address from a locally stored sending address set, wherein the address in the sending address set is the address in the second bilateral memory;

[0100] Writing the reply information corresponding to the data read / write request to the target sending bilateral memory address and submitting the reply information to the reply information queue;

[0101] Check whether the reply information meets the sending condition, and if the reply information meets the sending condition, send the reply information to the storage client on the second node by using the bilateral interface RDMA send.

[0102] In some specific implementations, the reading and writing module 13 is used to:

[0103] When the data read / write request is a data write request, an RDMA unilateral data read operation is performed on the memory corresponding to the unilateral memory address to obtain the data to be written, and the data to be written is written into the second unilateral memory.

[0104] In some specific implementations, the reading and writing module 13 is used to:

[0105] When the data read / write request is a data read request, the to-be-read data corresponding to the data read request is read from the storage device of the first node, and the to-be-read data is written to the memory corresponding to the unilateral memory address.

[0106] In some specific implementations, the data reading and writing device further includes:

[0107] The connection exception processing module is configured to release the first bilateral memory, the second bilateral memory, the second unilateral memory, and the connection resource when the RDMA connection is terminated.

[0108] For further information, see Figure 8 As shown, an embodiment of the present application further discloses an electronic device, including: a processor 21 and a memory 22.

[0109] The memory 22 is used to store computer programs; the processor 21 is used to execute the computer programs to implement the data reading and writing methods disclosed in the aforementioned embodiments.

[0110] For the specific process of the above-mentioned data reading and writing method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0111] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data reading and writing method disclosed in any of the aforementioned embodiments.

[0112] For the specific process of the above-mentioned data reading and writing method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0115] Finally, it should be noted that, in this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a series of processes, methods, articles or devices that include other elements include not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such processes, methods, articles or devices. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0116] The above is a detailed introduction to a data reading and writing method, device, equipment, and medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data reading and writing method, characterized in that: The storage service client applied to the first node in the distributed storage system includes: Receiving, using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, wherein the data read / write request is sent by the storage client on the second node using the bilateral interface RDMA send, and an RDMA connection established based on the RoCE protocol exists between the storage service client on the first node and the storage client on the second node; Obtaining a unilateral memory address from the data read / write request, wherein the unilateral memory address is an address in a first unilateral memory on the second node for storing read / write data transmitted between the first node and the second node through the RDMA connection; Performing an RDMA unilateral data read and write operation on the memory corresponding to the unilateral memory address to respond to the data read and write request; Sending reply information corresponding to the data read / write request to the storage client on the second node using the bilateral interface RDMA send, so that the storage client on the second node receives the reply information using the bilateral interface RDMA recv, wherein the reply information is used to indicate that the operation corresponding to the data read / write request has been completed; Correspondingly, before receiving the data read and write request sent by the storage client on the second node in the distributed storage system by using the bilateral interface RDMA recv, the method further includes: When the storage service client on the first node is initialized, applying for and registering a first bilateral memory and a second bilateral memory, and pre-registering a second unilateral memory; wherein the second unilateral memory is a memory in the first node used to store data to be written transmitted between the first node and the second node via the RDMA connection; When the storage service client on the first node receives the connection establishment request sent by the storage client on the second node, it creates connection resources corresponding to the RDMA connection and establishes the RDMA connection.

2. The data reading and writing method according to claim 1, wherein: The first bilateral memory is a memory in the first node used to store data received through the bilateral interface RDMA recv, and the second bilateral memory is a memory in the first node used to store data sent through the bilateral interface RDMA send.

3. The data reading and writing method according to claim 2, wherein: The receiving, by using the bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, includes: Retrieving a target receiving bilateral memory address from a locally stored receiving address set, wherein the address in the receiving address set is the address in the first bilateral memory; A data read / write request sent by a storage client on a second node in the distributed storage system is received using a bilateral interface RDMA recv, and information included in the data read / write request is stored in the target receiving bilateral memory address.

4. The data reading and writing method according to claim 2, wherein: The sending reply information corresponding to the data read and write request to the storage client on the second node by using the bilateral interface RDMA send includes: Retrieving a target sending bilateral memory address from a locally stored sending address set, wherein the address in the sending address set is the address in the second bilateral memory; Writing the reply information corresponding to the data read / write request to the target sending bilateral memory address and submitting the reply information to the reply information queue; Check whether the reply information meets the sending condition, and if the reply information meets the sending condition, send the reply information to the storage client on the second node by using the bilateral interface RDMA send.

5. The data reading and writing method according to claim 2, wherein: The performing RDMA unilateral data reading and writing operations on the memory corresponding to the unilateral memory address includes: When the data read / write request is a data write request, an RDMA unilateral data read operation is performed on the memory corresponding to the unilateral memory address to obtain the data to be written, and the data to be written is written into the second unilateral memory.

6. The data reading and writing method according to claim 2, characterized in that: The performing RDMA unilateral data reading and writing operations on the memory corresponding to the unilateral memory address includes: When the data read / write request is a data read request, the to-be-read data corresponding to the data read request is read from the storage device of the first node, and the to-be-read data is written to the memory corresponding to the unilateral memory address.

7. The data reading and writing method according to claim 2, characterized in that: Also includes: When the RDMA connection is destroyed, the first bilateral memory, the second bilateral memory, the second unilateral memory, and the connection resources are released.

8. A data reading and writing device, characterized in that: The storage service client applied to the first node in the distributed storage system includes: a request acquisition module, configured to receive, using a bilateral interface RDMA recv, a data read / write request sent by a storage client on a second node in the distributed storage system, wherein the data read / write request is sent by the storage client on the second node using the bilateral interface RDMA send, and an RDMA connection established based on the RoCE protocol exists between the storage service client on the first node and the storage client on the second node; an information extraction module, configured to obtain a unilateral memory address from the data read / write request, wherein the unilateral memory address is an address in a first unilateral memory on the second node for storing read / write data transmitted between the first node and the second node via the RDMA connection; A read / write module, configured to perform RDMA unilateral data read / write operations on the memory corresponding to the unilateral memory address to respond to the data read / write request; a reply information sending module, configured to send a reply message corresponding to the data read / write request to the storage client on the second node using a bilateral interface RDMA send, so that the storage client on the second node receives the reply message using a bilateral interface RDMA recv, wherein the reply message is used to indicate that the operation corresponding to the data read / write request has been completed; Correspondingly, before receiving the data read and write request sent by the storage client on the second node in the distributed storage system by using the bilateral interface RDMA recv, the method further includes: When the storage service client on the first node is initialized, applying for and registering a first bilateral memory and a second bilateral memory, and pre-registering a second unilateral memory; wherein the second unilateral memory is a memory in the first node used to store data to be written transmitted between the first node and the second node via the RDMA connection; When the storage service client on the first node receives the connection establishment request sent by the storage client on the second node, it creates connection resources corresponding to the RDMA connection and establishes the RDMA connection.

9. An electronic device, characterized in that: include: memory and processor; Wherein, the memory is used to store computer programs; The processor is configured to execute the computer program to implement the data reading and writing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the data reading and writing method according to any one of claims 1 to 7 is implemented.

Images