Memory access method and device, electronic equipment and storage medium

Abstract

The invention provides a memory access method and device, electronic equipment and a storage medium, and the method comprises the steps: determining registered memories of a first network card and a second network card, building connection between a client and a distributed server through the first network card and the second network card, responding to a remote access request of the server, obtaining a destination address corresponding to a target server, and sending the destination address to the server; determining a target node, connected with the client, of the target server and a registered memory of a network card corresponding to the target node, dividing the first data block into a preset number of target data blocks and erasure verification blocks, and copying the target data blocks and erasure verification blocks to the registered memory of the network card corresponding to the target node, and reading the target data block and the erasure verification block of the registered memory through the target server, and writing the second data block to be returned to the client into the registered memory. According to the invention, the network card is ensured to only operate the memory with good affinity, the network communication efficiency in distributed storage is improved, and the memory access effectiveness of the distributed storage system is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a memory access method, device, electronic device and storage medium. Background Art

[0002] The NUMA (Non Uniform Memory Access) architecture consists of multiple CPUs, each of which is independently connected to a part of the memory. The memory that some CPUs directly connect to through the internal bus is called local memory. The memory that the interconnection module between CPUs accesses and is not directly connected to is called remote memory. The speed of accessing local memory is much faster than remote memory. RDMA (Remote Direct Memory Access) is a direct memory access technology that realizes the direct transmission of data buffers between two nodes during network transmission. The local node can directly transmit data to the memory of the remote node through the network. Therefore, RDMA communication is widely used in high-performance distributed storage scenarios due to its advantages such as high bandwidth, low latency and kernel bypass.

[0003] At present, the RDMA RPC solution in high-performance distributed storage usually performs erasure calculations on the client side. When the RPC request is issued, the data may be located in the RDMA registered memory of the NUMA node or in the ordinary memory. The client first needs to copy the data once, copying the original data from the ordinary memory to the RDMA registered memory, ensuring that the RDMA network card can remotely operate the data, and send the data information and the erasure check block information obtained by the erasure calculation to the storage pool server, which completes the unilateral writing and reading of the data. However, in the case of multiple copies or erasure services, since the RDMA network card operates the registered memory of two or more NUMA nodes at the same time, there is a tailing phenomenon in the multi-NUMA scenario, which will cause the performance of RDMA communication to be severely attenuated, thereby affecting the RDMA communication efficiency of the high-performance distributed storage system. Summary of the invention

[0004] In view of this, the present invention aims to propose a memory access method, device, electronic device and storage medium to solve the problem that the RDMA network card simultaneously operates the registered memory of two or more NUMA nodes, affecting the RDMA communication efficiency of the high-performance distributed storage system, and realize effective RDMA memory access.

[0005] According to a first aspect of the present invention, a memory access method is provided, which is applied to a distributed storage system, wherein the distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, and the second node corresponds to a second network card, the method includes:

[0006] Determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card;

[0007] In response to a remote access request from a server, a destination address corresponding to a target server is obtained, and the destination address is used to determine a target node connected between the target server and the client, and a registered memory of a network card corresponding to the target node; wherein the target node is the first node or the second node;

[0008] Dividing the first data block into a preset number of target data blocks and erasure check blocks through the client, and copying the blocks to the registered memory of the network card corresponding to the target node;

[0009] The target data block and the erasure check block of the registration memory are read through the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

[0010] Optionally, determining the registered memory of the first network card and the second network card, and establishing a connection between the client and the distributed server through the first network card and the second network card includes:

[0011] Applying for the registration memory of the first network card and the second network card in advance through the client, and determining the registration memory addresses of the first network card and the second network card;

[0012] Obtain node identifiers of the first node and the second node;

[0013] Get the node identifiers of multiple distributed servers;

[0014] According to the node identifier of the server and the node identifier of the client, the client is connected to the server through the first network card and the second network card; wherein the node identifier of the server is consistent with the node identifier of the network card in the client connected to the server;

[0015] According to the node identifier, the connection information between the server and the network card in the client is generated, and the connection information is stored.

[0016] Optionally, in response to the remote access request of the server, obtaining a destination address corresponding to the target server, using the destination address to determine a target node connected to the target server and the client, and a registered memory of a network card corresponding to the target node, wherein the target node is the first node or the second node, includes:

[0017] In response to the remote access request from the server, obtain the destination address corresponding to the target server;

[0018] Acquire pre-stored connection information between the server and the network card in the client, traverse the connection information using the destination address, and determine that the target node connected between the target server and the client is the first node or the second node;

[0019] Determine the registered memory of the network card corresponding to the target node to which the target server and the client are connected.

[0020] Optionally, dividing the first data block into a preset number of target data blocks and erasure check blocks by the client, and copying the blocks to a registered memory of a network card corresponding to the target node, includes:

[0021] The client uses a preset erasure algorithm to divide the first data block of the client into a preset number of target data blocks and a preset number of erasure check blocks;

[0022] Obtaining the storage address of the target data block;

[0023] According to the storage address, the target data block and the erasure check block are respectively copied to the registered memory of the network card corresponding to the target node.

[0024] Optionally, the copying, according to the storage address, the target data block and the erasure check block to a registered memory of a network card corresponding to the target node respectively includes:

[0025] If the storage address of the target data block is a common memory, copying the target data block and the erasure check block to the registered memory of the network card corresponding to the target node;

[0026] If the storage address of the target data block is a registered memory, it is determined that the first data block is stored in the registered memory of the network card corresponding to the target node.

[0027] Optionally, the target server includes a main server node and a sub-server node, and the target data block and the erasure check block of the registration memory are read through the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node, including:

[0028] The client sends metadata information of the erasure check block to the main server node, and the main server node forwards the metadata information to each sub-server node;

[0029] Reading the target data block and the erasure check block of the registration memory respectively according to the metadata information by the main server node and the sub server node;

[0030] It is monitored that the main server node and the sub-server node have completed reading, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

[0031] Optionally, after reading the target data block and the erasure check block from the registration memory through the target server and writing the second data block to be returned to the client into the registration memory of the network card corresponding to the target node, the method further includes:

[0032] Obtaining the total number of data blocks read by the main server node and the sub-server node, and determining whether the total number of data blocks is greater than the target number of data blocks;

[0033] If the total number of data blocks is greater than or equal to the target number of data blocks, a read success message is returned to the client;

[0034] If the total number of data blocks is less than the target number of data blocks, the read failure data is returned to the client.

[0035] According to a second aspect of the present invention, a memory access device is provided, which is applied to a distributed storage system, wherein the distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, and the second node corresponds to a second network card, and the device includes:

[0036] Determine a memory module, which is used to determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card;

[0037] A node determination module, used to respond to the remote access request of the server, obtain the destination address corresponding to the target server, and use the destination address to determine the target node connected to the target server and the client, and the registration memory of the network card corresponding to the target node; wherein the target node is the first node or the second node;

[0038] A data erasure module, used for dividing the first data block into a preset number of target data blocks and erasure check blocks through the client, and copying the blocks to the registration memory of the network card corresponding to the target node;

[0039] The data reading and writing module is used to read the target data block and the erasure check block in the registration memory through the target server, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

[0040] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0041] processor;

[0042] a memory for storing instructions executable by the processor;

[0043] The processor is configured to execute the instructions to implement the memory access method as described above.

[0044] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the memory access method described above are implemented.

[0045] The memory access method provided by the embodiment of the present invention determines the registered memory of the first network card and the second network card, and establishes a connection between the client and the distributed server through the first network card and the second network card, obtains the destination address corresponding to the target server in response to the remote access request of the server, uses the destination address to determine the target node connected to the target server and the registered memory of the network card corresponding to the target node, divides the first data block into a preset number of target data blocks and erasure check blocks through the client, and copies them to the registered memory of the network card corresponding to the target node, reads the target data blocks and erasure check blocks in the registered memory through the target server, and writes the second data block to be returned to the client into the registered memory of the network card corresponding to the target node. The embodiment of the present invention is applicable to RDMA communication scenarios under high-performance distributed storage, optimizes the communication efficiency of RDMA, establishes a connection between a client network card and a server according to a NUMA node, uses the NUMA node information of the client network card, applies for RDMA registered memory under the corresponding NUMA node according to the NUAM node information during erasure calculation, copies the original data to the applied multiple RDMA registered memories, ensures that the RDMA network card only operates the memory with the best affinity to achieve the purpose of communication optimization, can improve the network communication efficiency in distributed storage, reduce the tail delay problem of distributed services, and further improve the memory access effectiveness of the distributed storage system.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1is a flowchart of a memory access method provided by an embodiment of the present invention;

[0049] Figure 2 yes Figure 1 A flowchart of step 101 in the memory access method provided by an embodiment of the present invention;

[0050] Figure 3 yes Figure 1 A flowchart of step 102 in the memory access method provided by an embodiment of the present invention;

[0051] Figure 4 yes Figure 1 A flowchart of step 103 in the memory access method provided by an embodiment of the present invention;

[0052] Figure 5 yes Figure 1 A flowchart of step 104 in the memory access method provided by an embodiment of the present invention;

[0053] Figure 6 is a flowchart of another memory access method provided by an embodiment of the present invention;

[0054] Figure 7 is a schematic diagram of a scenario of a memory access method provided by an embodiment of the present invention;

[0055] Figure 8 is a structural diagram of a memory access device provided by an embodiment of the present invention;

[0056] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0058] Reference Figure 1, shows a flowchart of the steps of a memory access method provided by an embodiment of the present invention, which is applied to a distributed storage system, wherein the distributed storage system includes a client and multiple distributed servers, wherein the client includes a first node and a second node, wherein the first node corresponds to a first network card, and the second node corresponds to a second network card, wherein the method may include:

[0059] Step 101: determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card.

[0060] In an embodiment of the present invention, a distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, and the second node corresponds to a second network card. When the client process is started, it first checks the status of the two network cards to ensure that both network cards are in an available state, and registers two memory areas through the RDMA API, binds them to the two RDMA network cards respectively, and determines the registered memory of the first network card and the second network card. When multiple distributed servers are started, according to the NUMA node information of the server's own NVME disk, select the client RDMA network card associated with it, to ensure that one network card only operates the RDMA registered memory of one NUMA node, so as to optimize the data transmission path and ensure that the RDMA performance degradation caused by bus competition is avoided.

[0061] Specifically, the distributed storage system applies for the registered memory of the first network card and the second network card in advance through the client, determines the registered memory addresses of the first network card and the second network card, respectively obtains the node identifiers of the first node and the second node, and the node identifiers of multiple distributed servers, so as to establish a connection between the client and the server through the first network card and the second network card according to the node identifier of the server and the node identifier of the client; wherein the node identifier of the server is consistent with the node identifier of the network card in the client to which the server is connected, and according to the node identifier, generates connection information between the server and the network card in the client, and stores the connection information.

[0062] Step 102, in response to the remote access request of the server, obtain the destination address corresponding to the target server, use the destination address to determine the target node connected to the target server and the client, and the registration memory of the network card corresponding to the target node.

[0063] Wherein, the target node is the first node or the second node;

[0064] In an embodiment of the present invention, in response to a remote access request from a server, a destination address corresponding to a target server is obtained, pre-stored connection information between the server and a network card in a client is obtained, the connection information is traversed using the destination address, it is determined that the target node connected to the target server and the client is the first node or the second node, and the registered memory of the network card corresponding to the target node connected to the target server and the client is determined.

[0065] Specifically, when the client receives a remote access request RPC request from the server, a distributed consistency algorithm can be used to calculate the destination address corresponding to the target server, wherein the target server is a server that needs to access the client memory, and the target server includes a main server node and a sub-server node. The pre-stored connection information between the server and the network card in the client is obtained, and the connection information is traversed using the destination address to determine whether the target node connected to the target server and the client is the first node or the second node. Specifically, the client queries the connection relationship in the stored server map and obtains the NUMA node information corresponding to the target server address in order to select a suitable RDMA registered memory. The client selects the RDMA registered memory corresponding to the server address based on the NUMA node information.

[0066] Step 103: The first data block is divided into a preset number of target data blocks and erasure check blocks by the client, and the blocks are copied to the registered memory of the network card corresponding to the target node.

[0067] In an embodiment of the present invention, the first data block is divided into a preset number of target data blocks and a preset number of erasure check blocks by the client. In this embodiment, the erasure check blocks are generally used to improve the reliability and fault tolerance of data. By generating additional check blocks, even if part of the data blocks are lost, the data can be restored through the check blocks. The client obtains the storage address of the target data block. The storage address of the target data block is the location of the target data block in the memory, or the location where the target data block is to be stored. According to the storage address, the client copies the target data block and the erasure check block to the registered memory of the network card corresponding to the target node, respectively, to ensure that the data block and the erasure check block are correctly stored in the registered memory of the target node for subsequent data transmission and processing. By using the erasure algorithm, the reliability and fault tolerance of the data are improved, and the risk of data loss is reduced.

[0068] Step 104 , read the target data block and the erasure check block in the registration memory through the target server, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

[0069] In an embodiment of the present invention, metadata information of an erasure check block is sent to the main server node through a client, and the main server node is used to forward the metadata information to each sub-server node, and the main server node and the sub-server node read the target data block and the erasure check block of the registered memory according to the metadata information respectively, and when the main server node and the sub-server node are monitored to complete the reading, the second data block to be returned to the client is written into the registered memory of the network card corresponding to the target node.

[0070] The memory access method provided by the embodiment of the present invention determines the registered memory of the first network card and the second network card, and establishes a connection between the client and the distributed server through the first network card and the second network card, obtains the destination address corresponding to the target server in response to the remote access request of the server, uses the destination address to determine the target node connected to the target server and the registered memory of the network card corresponding to the target node, divides the first data block into a preset number of target data blocks and erasure check blocks through the client, and copies them to the registered memory of the network card corresponding to the target node, reads the target data blocks and erasure check blocks in the registered memory through the target server, and writes the second data block to be returned to the client into the registered memory of the network card corresponding to the target node. The embodiment of the present invention is applicable to RDMA communication scenarios under high-performance distributed storage, optimizes the communication efficiency of RDMA, establishes a connection between a client network card and a server according to a NUMA node, uses the NUMA node information of the client network card, applies for RDMA registered memory under the corresponding NUMA node according to the NUAM node information during erasure calculation, copies the original data to the applied multiple RDMA registered memories, ensures that the RDMA network card only operates the memory with the best affinity to achieve the purpose of communication optimization, can improve the network communication efficiency in distributed storage, reduce the tail delay problem of distributed services, and further improve the memory access effectiveness of the distributed storage system.

[0071] In order to facilitate those skilled in the art to fully understand the above memory access method, refer to Figure 7 , Figure 7It is a scenario diagram of a memory access method provided by an embodiment of the present invention, and takes a single client node with two RDMA network cards, a storage pool process with one RDMA network card, an NVME disk, and a distributed storage pool 4+2 erasure as an example for explanation. Specifically: a distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, the second node corresponds to a second network card, a client process and six storage pool server processes, the client node uses two network cards at the same time, which are respectively located at different NUMA nodes, namely the first node and the second node, and to ensure that one network card only operates the RDMA registered memory of one NUMA node, a connection between the client and the distributed server is established through the first network card and the second network card, and the server process can select the client network card connection with the same CPU as the NVME according to the identifier of the NUMA node of the NVME non-volatile memory host controller disk to avoid cross-CPU communication of data flow. It should be noted that, taking the above figure as an example, the data blocks 0, 1, 2, 3 on the client are converted into memory blocks 0, 1, 2, 3, 4, 5 after erasure calculation, where 0, 1, 2, 3 are original data blocks, 4, 5 are erasure check blocks, data blocks 0, 2, 4 are RDMA communication through the second network card of the second node, and the RDMA registration memory of the second node is used, data blocks 1, 3, 5 are RDMA communication through the first network card of the first node, and the RDMA registration memory of the second node is used, and the server is evenly connected to the first network card and the second network card of the client according to the node identifier. Of course, the above is a specific example. In actual operation, the distributed storage system may include more client nodes and network cards, as well as a storage pool server, and this embodiment does not specifically limit this.

[0072] Further, see Figure 2 , showing Figure 1 A flowchart of step 101 in a memory access method is provided. The method is substantially the same as the memory access method provided in the first embodiment of the present invention. Step 101 may include:

[0073] Step 201: Apply for the registration memory of the first network card and the second network card in advance through the client, and determine the registration memory addresses of the first network card and the second network card.

[0074] Step 202: Obtain node identifiers of the first node and the second node.

[0075] Step 203: Obtain node identifiers of multiple distributed servers.

[0076] Step 204, according to the node identifier of the server and the node identifier of the client, establish a connection between the client and the server through the first network card and the second network card; wherein the node identifier of the server is consistent with the node identifier of the network card in the client connected to the server.

[0077] Step 205: Generate connection information between the server and the network card in the client according to the node identifier, and store the connection information.

[0078] In an embodiment of the present invention, the client pre-applies for a registered memory for the first network card and the second network card, determines the registered memory addresses of the first network card and the second network card, and obtains the node identifiers of the first node and the second node. The node identifier is used to distinguish different NUMA nodes and obtain the node identifiers of multiple distributed servers. It is used to determine the location and attributes of the server, and according to the node identifier of the server and the node identifier of the client, the client is connected to the server through the first network card and the second network card, ensuring that the node identifier of the server is consistent with the node identifier of the network card in the client connected to the server, so as to optimize the data transmission path and avoid communication across NUMA nodes or CPU cores. According to the node identifier, the connection information between the server and the network card in the client is generated, and the connection information is stored for subsequent query and use. By ensuring that the node identifiers of the server and the client are consistent, the communication across NUMA nodes or CPU cores is reduced, thereby reducing the delay and improving the data transmission efficiency.

[0079] It should be noted that the client can use a consistent hashing algorithm to map the server's node identifier to two RDMA network cards to ensure load balancing. The client generates connection information between the server and the network card in the client, where the connection information includes the server's node identifier, RDMA network card information, and NUMA node information, and stores the connection information in the server map for quick query.

[0080] Further, see Figure 3 , showing Figure 1 A flowchart of step 102 in a memory access method is provided. The method is substantially the same as the memory access method provided in the first embodiment of the present invention. Step 102 may include:

[0081] Step 301, in response to a remote access request from a server, obtain a destination address corresponding to a target server.

[0082] Step 302, obtaining pre-stored connection information between the server and the network card in the client, traversing the connection information using the destination address, and determining whether the target node connected between the target server and the client is the first node or the second node.

[0083] Step 303, determine the registered memory of the network card corresponding to the target node to which the target server and the client are connected.

[0084] It should be noted that, in an embodiment of the present invention, when a client receives a remote access request RPC request from a server, a distributed consistency algorithm can be used to calculate a destination address corresponding to a target server, wherein the target server is a server that needs to access the client memory, and the target server includes a main server node and a sub-server node. The pre-stored connection information between the server and the network card in the client is obtained, and the connection information is traversed using the destination address to determine whether the target node connected to the target server and the client is the first node or the second node. Specifically, the client queries the connection relationship in the stored server map to obtain the NUMA node information corresponding to the target server address so as to select a suitable RDMA registered memory. The client selects the RDMA registered memory corresponding to the server address based on the NUMA node information.

[0085] In an embodiment of the present invention, when a client receives a remote access request from a server, the client obtains a destination address corresponding to a target server, where the destination address is an IP address or a network identifier of the server. The client obtains pre-stored connection information between the server and the network card in the client, and uses the destination address to traverse the connection information to determine whether the target node connected to the target server and the client is the first node or the second node, thereby ensuring optimization of the data transmission path and avoiding communication across NUMA nodes or CPU cores. Based on the determined target node (the first node or the second node), the client determines the registered memory of the network card corresponding to the target node connected to the target server and the client, thereby ensuring that the correct memory area is used during data transmission, thereby improving the efficiency and reliability of memory access.

[0086] Further, see Figure 4 , showing Figure 1 A flowchart of step 103 in a memory access method is provided. The method is substantially the same as the memory access method provided in the first embodiment of the present invention. Step 103 may include:

[0087] Step 401: The client uses a preset erasure algorithm to divide a first data block of the client into a preset number of target data blocks and a preset number of erasure check blocks.

[0088] In an embodiment of the present invention, a client uses a preset erasure algorithm to divide the first data block of the client into a preset number of target data blocks and a preset number of erasure check blocks. Specifically, in a distributed storage system, the computation and storage tasks of the erasure code are distributed to multiple nodes to improve the scalability and fault tolerance of the system. The erasure code parameters can be adjusted dynamically, that is, the parameters K (number of data blocks) and M (number of check blocks) of the erasure code can be adjusted dynamically. For example, for critical data, the M value can be increased to improve fault tolerance; for frequently accessed data, the K value can be reduced to improve reading efficiency. In this embodiment, the erasure algorithm can be adaptively selected or adjusted according to the real-time status of the system (such as network bandwidth, storage load, failure rate, etc.), which is not specifically limited here.

[0089] Step 402: Obtain the storage address of the target data block.

[0090] Step 403: According to the storage address, the target data block and the erasure check block are copied to the registered memory of the network card corresponding to the target node respectively.

[0091] It should be noted that, in an embodiment of the present invention, the client uses a preset erasure algorithm to divide the first data block into a preset number of target data blocks and a preset number of erasure check blocks. In this embodiment, the erasure algorithm is generally used to improve the reliability and fault tolerance of data. By generating additional check blocks, even if part of the data blocks are lost, the data can be restored through the check blocks, and the client obtains the storage address of the target data block. These addresses may be the location of the target data block in the memory, or the location where the target data block is to be stored. According to the storage address, the client copies the target data block and the erasure check block to the registered memory of the network card corresponding to the target node, respectively, to ensure that the data block and the erasure check block are correctly stored in the registered memory of the target node for subsequent data transmission and processing. By using the erasure algorithm, the reliability and fault tolerance of the data are improved, and the risk of data loss is reduced.

[0092] Specifically, the copying of the target data block and the erasure check block to the registered memory of the network card corresponding to the target node according to the storage address includes:

[0093] If the storage address of the target data block is a common memory, copying the target data block and the erasure check block to the registered memory of the network card corresponding to the target node;

[0094] If the storage address of the target data block is a registered memory, it is determined that the first data block is stored in the registered memory of the network card corresponding to the target node.

[0095] It should be noted that in the above steps, based on the storage address of the target data block, it is determined whether to copy the target data block and the erasure check block to the registered memory of the network card corresponding to the target node. If the storage address of the target data block is ordinary memory, the target data block and the erasure check block are copied to the registered memory of the network card corresponding to the target node, wherein the ordinary memory is a memory area of ​​the client that is not a registered memory, which is not specifically limited here.

[0096] Specifically, the client applies for the RDMA registered memory of the network card according to the obtained NUMA node information, and the client uses the erasure data algorithm to divide the original data block into K target data blocks and M erasure check blocks. For example, data blocks 0, 1, 2, 3 on the client are converted into 0, 1, 2, 3, 4, 5 memory blocks after erasure calculation, wherein 0, 1, 2, 3 are original data blocks, 4, 5 are erasure check blocks, data blocks 0, 2, 4 perform RDMA communication through the second network card of the second node, and use the RDMA registered memory of the second node, data blocks 1, 3, 5 perform RDMA communication through the first network card of the first node, and use the RDMA registered memory of the second node. According to the storage address of the data block, the target data block and the erasure check block are copied to the registered memory of the network card corresponding to the target node respectively. Specifically, the data is copied to the selected RDMA registered memory. If the storage address of the target data block is the registered memory, that is, the target data block is already in the RDMA registered memory, then the erasure check block does not need to be copied. If the storage address of the target data block is the ordinary memory, all the data needs to be copied, and the target data block and the erasure check block are copied to the registered memory of the network card corresponding to the target node.

[0097] Further, see Figure 5 , showing Figure 1 A flowchart of step 104 in a memory access method is provided, which is substantially the same as the memory access method provided in the first embodiment of the present invention, wherein the target server includes a main server node and a sub-server node, and step 104 may include:

[0098] Step 501: Send metadata information of the erasure check block to the main server node through the client, and use the main server node to forward the metadata information to each sub-server node.

[0099] Step 502: read the target data block and the erasure check block of the registered memory according to the metadata information respectively through the main server node and the sub-server node.

[0100] Step 503: When the master server node and the slave server node are monitored to complete reading, the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

[0101] It should be noted that, in an embodiment of the present invention, the target server includes a main server node and a sub-server node, specifically a main server node and multiple sub-server nodes. The client sends metadata information of the erasure check block to the main server node, and the main server node forwards the metadata information to each sub-server node. The metadata information includes the address, size, check value and other information of the erasure check block. The main server node and the sub-server node read the target data block and the erasure check block from the registered memory according to the received metadata information, respectively, to ensure that the server can correctly obtain the data block and check block sent by the client. When the main server node and the sub-server node complete the reading operation, they write the second data block to be returned to the client into the registered memory of the network card corresponding to the target node, to ensure that the client can receive the data block returned by the server.

[0102] Specifically, the client assembles metadata information of multiple erasure data blocks into a message and sends it to the main server node through the RDMA send primitive operation, including the address and key value. The main server node forwards the metadata information to each sub-server node through the RDMA send operation primitive. The main server node reads the client's business message through the unilateral read operation primitive according to the address information and key value of the metadata information, writes the local data to the client's RDMA registered memory address through the unilateral write operation, and saves the result; the sub-server node reads the client's business message through the unilateral read operation primitive according to the address information and key value of the metadata information, writes the local data to the client's RDMA registered memory address through the unilateral write operation. It should be noted that after the sub-server node completes the read and write operations, it feeds back the results to the main server node through the RDMAsend operation primitive. After the main server node waits for the unilateral operation results of this node and all sub-nodes to be completed, it returns the second data block to be returned to the client through the RDMA send operation primitive.

[0103] Reference Figure 6 , shows a flow chart of steps of another memory access method provided by an embodiment of the present invention, which method is basically the same as the memory access method provided by the first embodiment of the present invention, except that the method may further include:

[0104] Step 101: determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card.

[0105] Step 102, in response to the remote access request of the server, obtain the destination address corresponding to the target server, use the destination address to determine the target node connected to the target server and the client, and the registration memory of the network card corresponding to the target node.

[0106] Step 103: The first data block is divided into a preset number of target data blocks and erasure check blocks by the client, and the blocks are copied to the registered memory of the network card corresponding to the target node.

[0107] Step 104 , read the target data block and the erasure check block in the registration memory through the target server, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

[0108] The above steps 101 to 104 are described in the foregoing description and will not be repeated here.

[0109] Step 105, obtaining the total number of data blocks read by the main server node and the sub-server node, and determining whether the total number of data blocks is greater than the target number of data blocks.

[0110] Step 106: If the total number of data blocks is greater than or equal to the target number of data blocks, a read success message is returned to the client.

[0111] Step 107: If the total number of data blocks is less than the target number of data blocks, return the read failure data to the client.

[0112] It should be noted that, in the embodiment of the present invention, after the target server reads the target data block and the erasure check block of the registration memory, and writes the second data block to be returned to the client into the registration memory of the network card corresponding to the target node, the system obtains the total number of data blocks read by the main server node and the sub-server node, and determines whether the total number of data blocks is greater than the target number of data blocks. Specifically, after the main server node waits for the unilateral operation results of the node and the unilateral operation results of all sub-nodes to be completed, it determines whether the number of completed data blocks is greater than the operation erasure K value of the target number of data blocks. If the total number of data blocks is greater than or equal to the target number of data blocks, a read success message is returned to the client. If the total number of data blocks is less than the target number of data blocks, a read failure data is returned to the client, that is, if it is greater than or equal to K, it is successful, and if it is less than K, it represents failure. The result is returned to the client through the RDMA send operation primitive. The client determines whether the service request needs to be resent according to the result of the main service feedback. If the feedback is successful, the entire RPC process ends.

[0113] Reference Figure 8 , shows a schematic diagram of the structure of a memory access device provided by an embodiment of the present invention, which is applied to a distributed storage system, wherein the distributed storage system includes a client and multiple distributed servers, wherein the client includes a first node and a second node, wherein the first node corresponds to a first network card, and the second node corresponds to a second network card, wherein the device includes:

[0114] Determine a memory module 601, which is used to determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card;

[0115] The node determination module 602 is used to obtain the destination address corresponding to the target server in response to the remote access request of the server, and use the destination address to determine the target node connected to the target server and the client, and the registered memory of the network card corresponding to the target node; wherein the target node is the first node or the second node;

[0116] A data erasure module 603, configured to divide the first data block into a preset number of target data blocks and erasure check blocks through the client, and copy the blocks to the registration memory of the network card corresponding to the target node;

[0117] The data reading and writing module 604 is used to read the target data block and the erasure check block in the registration memory through the target server, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

[0118] Furthermore, the memory determination module 601 includes:

[0119] A first determining submodule, configured to apply for the registration memory of the first network card and the second network card in advance through the client, and determine the registration memory addresses of the first network card and the second network card;

[0120] A first acquisition submodule, used to acquire node identifiers of the first node and the second node;

[0121] The second acquisition submodule is used to obtain node identifiers of multiple distributed servers;

[0122] A connection submodule, used to establish a connection between the client and the server through the first network card and the second network card according to the node identifier of the server and the node identifier of the client; wherein the node identifier of the server is consistent with the node identifier of the network card in the client connected to the server;

[0123] A generating submodule is used to generate connection information between the server and the network card in the client according to the node identifier, and store the connection information.

[0124] Furthermore, the node determination module 602 includes:

[0125] A third acquisition submodule is used to obtain a destination address corresponding to a target server in response to a remote access request from the server;

[0126] A second determination submodule is used to obtain pre-stored connection information between the server and the network card in the client, traverse the connection information using the destination address, and determine a target node for connection between the target server and the client;

[0127] The third determination submodule is used to determine the registered memory of the network card corresponding to the target node connected to the target server and the client.

[0128] Furthermore, the data erasure module 603 includes:

[0129] A division submodule, configured to divide the first data block of the client into a preset number of target data blocks and a preset number of erasure check blocks by the client using a preset erasure algorithm;

[0130] A third acquisition submodule is used to acquire the storage address of the target data block;

[0131] The copy submodule is used to copy the target data block and the erasure check block to the registered memory of the network card corresponding to the target node according to the storage address.

[0132] Furthermore, the copy submodule 604 includes:

[0133] A copy unit, configured to copy the target data block and the erasure check block to a registered memory of a network card corresponding to the target node if the storage address of the target data block is a common memory;

[0134] A determining unit is configured to determine, if the storage address of the target data block is a registered memory, that the first data block is stored in the registered memory of the network card corresponding to the target node.

[0135] Further, the target server includes a main server node and a sub-server node, and the data reading and writing module includes:

[0136] A sending submodule, used to send metadata information of the erasure check block to the main server node through the client, and use the main server node to forward the metadata information to each sub-server node;

[0137] A reading submodule, used for reading the target data block and the erasure check block of the registration memory respectively according to the metadata information through the main server node and the sub server node;

[0138] The writing submodule is used to detect that the reading of the main server node and the sub-server node is completed, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

[0139] Furthermore, the device also includes:

[0140] A data block determination module is used to obtain the total number of data blocks read by the main server node and the sub-server node, and determine whether the total number of data blocks is greater than the target number of data blocks;

[0141] A first message module, configured to return a read success message to the client if the total number of data blocks is greater than or equal to the target number of data blocks;

[0142] The second message module is used to return read failure data to the client if the total number of data blocks is less than the target number of data blocks.

[0143] The memory access device provided by the embodiment of the present invention determines the registered memory of the first network card and the second network card, establishes a connection between the client and the distributed server through the first network card and the second network card, obtains the destination address corresponding to the target server in response to the remote access request of the server, uses the destination address to determine the target node connected to the target server and the registered memory of the network card corresponding to the target node, divides the first data block into a preset number of target data blocks and erasure check blocks through the client, and copies them to the registered memory of the network card corresponding to the target node, reads the target data blocks and erasure check blocks in the registered memory through the target server, and writes the second data block to be returned to the client into the registered memory of the network card corresponding to the target node. The embodiment of the present invention is applicable to RDMA communication scenarios under high-performance distributed storage, optimizes the communication efficiency of RDMA, establishes a connection between a client network card and a server according to a NUMA node, uses the NUMA node information of the client network card, applies for RDMA registered memory under the corresponding NUMA node according to the NUAM node information during erasure calculation, copies the original data to the applied multiple RDMA registered memories, ensures that the RDMA network card only operates the memory with the best affinity to achieve the purpose of communication optimization, can improve the network communication efficiency in distributed storage, reduce the tail delay problem of distributed services, and further improve the memory access effectiveness of the distributed storage system.

[0144] Reference Fig. 9 , an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0145] Memory 703, used for storing computer programs;

[0146] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0147] Determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card;

[0148] In response to a remote access request from a server, a destination address corresponding to a target server is obtained, and the destination address is used to determine a target node connected between the target server and the client, and a registered memory of a network card corresponding to the target node; wherein the target node is the first node or the second node;

[0149] Dividing the first data block into a preset number of target data blocks and erasure check blocks through the client, and copying the blocks to the registered memory of the network card corresponding to the target node;

[0150] The target data block and the erasure check block of the registration memory are read through the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

[0151] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0152] The communication interface is used for communication between the above terminal and other devices.

[0153] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0154] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0155] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the memory access method described in any one of the above embodiments.

[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0157] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A memory access method, characterized in that: Applied to a distributed storage system, the distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, and the second node corresponds to a second network card, the method includes: Determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card; In response to a remote access request from a server, a destination address corresponding to a target server is obtained, and the destination address is used to determine a target node connected between the target server and the client, and a registered memory of a network card corresponding to the target node; wherein the target node is the first node or the second node; Dividing the first data block into a preset number of target data blocks and erasure check blocks through the client, and copying the blocks to the registered memory of the network card corresponding to the target node; The target data block and the erasure check block of the registration memory are read through the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

2. The method according to claim 1, characterized in that The determining the registered memory of the first network card and the second network card, and establishing a connection between the client and the distributed server through the first network card and the second network card, includes: Applying for the registration memory of the first network card and the second network card in advance through the client, and determining the registration memory addresses of the first network card and the second network card; Obtain node identifiers of the first node and the second node; Get the node identifiers of multiple distributed servers; According to the node identifier of the server and the node identifier of the client, the client is connected to the server through the first network card and the second network card; wherein the node identifier of the server is consistent with the node identifier of the network card in the client connected to the server; According to the node identifier, the connection information between the server and the network card in the client is generated, and the connection information is stored.

3. The method according to claim 1, characterized in that The method of responding to a remote access request from a server, obtaining a destination address corresponding to a target server, and using the destination address to determine a target node connected to the target server and a registered memory of a network card corresponding to the target node includes: In response to the remote access request from the server, obtain the destination address corresponding to the target server; The pre-stored connection information between the server and the network card in the client is obtained, the connection information is traversed using the destination address, and it is determined that the target node connected between the target server and the client is the first node or the second node.

4. The method according to claim 1, characterized in that: The step of dividing the first data block into a preset number of target data blocks and erasure check blocks by the client, and copying the blocks to the registered memory of the network card corresponding to the target node includes: The client uses a preset erasure algorithm to divide the first data block of the client into a preset number of target data blocks and a preset number of erasure check blocks; Obtaining the storage address of the target data block; According to the storage address, the target data block and the erasure check block are respectively copied to the registered memory of the network card corresponding to the target node.

5. The method according to claim 4, characterized in that The step of copying the target data block and the erasure check block to the registered memory of the network card corresponding to the target node according to the storage address includes: If the storage address of the target data block is a common memory, copying the target data block and the erasure check block to the registered memory of the network card corresponding to the target node; If the storage address of the target data block is a registered memory, it is determined that the first data block is stored in the registered memory of the network card corresponding to the target node.

6. The method according to claim 1, characterized in that The target server includes a main server node and a sub-server node, and the target data block and the erasure check block of the registration memory are read through the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node, including: The client sends metadata information of the erasure check block to the main server node, and the main server node forwards the metadata information to each sub-server node; Reading the target data block and the erasure check block of the registration memory respectively according to the metadata information by the main server node and the sub server node; It is monitored that the main server node and the sub-server node have completed reading, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node.

7. The method according to claim 1, characterized in that After the target data block and the erasure check block of the registration memory are read by the target server, and the second data block to be returned to the client is written into the registration memory of the network card corresponding to the target node, the method further includes: Obtaining the total number of data blocks read by the main server node and the sub-server node, and determining whether the total number of data blocks is greater than the target number of data blocks; If the total number of data blocks is greater than or equal to the target number of data blocks, a read success message is returned to the client; If the total number of data blocks is less than the target number of data blocks, the read failure data is returned to the client.

8. A memory access device, characterized in that: Applied to a distributed storage system, the distributed storage system includes a client and multiple distributed servers, the client includes a first node and a second node, the first node corresponds to a first network card, the second node corresponds to a second network card, the device includes: Determine a memory module, which is used to determine the registered memory of the first network card and the second network card, and establish a connection between the client and the distributed server through the first network card and the second network card; A node determination module, used to respond to the remote access request of the server, obtain the destination address corresponding to the target server, and use the destination address to determine the target node connected to the target server and the client, and the registration memory of the network card corresponding to the target node; wherein the target node is the first node or the second node; A data erasure module, used for dividing the first data block into a preset number of target data blocks and erasure check blocks through the client, and copying the blocks to the registration memory of the network card corresponding to the target node; The data reading and writing module is used to read the target data block and the erasure check block in the registration memory through the target server, and write the second data block to be returned to the client into the registration memory of the network card corresponding to the target node.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the memory access method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the memory access method according to any one of claims 1 to 7 is implemented.

Images