Data access method and related device

Abstract

The present application provides a data access method and related device. The method includes a step of a network device receiving access requests sent by multiple clients connected to the network device, and sending the access requests to an access queue of a storage unit. The storage unit executes the access requests in the access queue and returns the processing results of the access requests of the multiple clients. The network device returns the processing results of the access requests returned by the storage unit to the client corresponding to the access requests. According to the method, the access requests sent by the multiple clients can be processed using the access queue, so that one access queue serves multiple clients. This breaks the inherent quantity limit of the access queue, supports large-scale networking connections, and expands the application scenarios.

Description

[Technical field]

[0001] The present invention relates to the field of storage technology, and in particular to data access methods and related devices. [Background technology]

[0002] With the rapid development of computer information technologies such as big data, cloud computing, and artificial intelligence in recent years, the scale of global Internet data has grown exponentially. Many high-concurrency and low-latency applications require high-performance hardware, which is why high-performance memory has emerged. In the case of high-performance memory, the distributed file system needs to allocate a large amount of computing resources to complete data processing and data exchange because of the powerful I / O throughput capacity of high-performance memory. As a result, the transmission latency of the system increases, limiting the network transmission capacity and system performance. To solve this problem, remote direct memory access (RDMA) has emerged. RDMA is a technology for direct remote memory access. Specifically, data can be moved directly and quickly from one system to another remote system memory without affecting the operating system. This reduces the consumption of the central processing unit (CPU) in the data transmission process, frees up memory bandwidth, and improves the service processing performance of the system. RDMA is characterized by high bandwidth, low latency, and low CPU usage.

[0003] Currently, when data is read and written using RDMA, the host's network device must first perform an RDMA operation to write data to the storage device's memory, and then the CPU in the storage device must save the data in the memory to a persistent storage medium, such as a solid state disk (SSD). However, using the CPU to save the data in the memory to a persistent storage medium requires consuming CPU resources. As a result, communication between the host and the storage device is affected. In addition, the submission queue (SQ) and completion queue (CQ) resources of the SSD are limited, so the storage device can only support a small number of network device connections, but cannot support a large number of network device connections.

[0004] Therefore, how to directly store data to persistent storage media by host network devices in large-scale networking connection scenarios and reduce the CPU usage of storage devices has become an urgent problem to be solved at present. Summary of the Invention

[0005] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention discloses a data access method and related apparatus to reduce the CPU usage of storage devices and expand applicable scenarios by persistently storing data directly across large-scale networking connections. [Means for solving the problem]

[0006] According to a first aspect, the present application provides a storage device, including a network device and a storage unit, the storage unit is connected to a plurality of clients using the network device, the network device is configured to send access requests of the plurality of clients to an access queue of the storage unit, the storage unit is configured to execute the access requests in the access queue and return a processing result of the access requests, and the network device is further configured to return the processing result of the access requests returned by the storage unit to the client corresponding to the access requests.

[0007] Optionally, the network device may be a network interface controller RNIC supporting remote direct memory access, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC) chip, or the like.

[0008] In the solution provided in this application, a network device sends access requests of multiple clients to an access queue for processing, and returns the processing results of the access requests to the corresponding clients, so that one access queue serves multiple clients, which breaks the inherent quantity limit of an access queue, supports large-scale networking connections, and expands application scenarios.

[0009] Referring to the first aspect, in one possible implementation of the first aspect, the network device stores a correspondence between information about multiple clients and an access queue, and the network device is configured to send access requests of the multiple clients to the access queue of the storage unit based on the correspondence.

[0010] In the solution provided in this application, the network device pre-stores the correspondence between the information about the client and the access queue, and sends the access requests of multiple clients to the access queue according to the correspondence, thus ensuring that the access queue can process the requests of multiple clients and the storage device can support large-scale networking connections.

[0011] Referring to the first aspect, in one possible implementation of the first aspect, the information about the multiple clients is connection information generated when the multiple clients respectively establish connections to the network device. When receiving an access request from any one of the multiple clients, the network device is configured to determine an access queue based on the connection information and the correspondence corresponding to the client carried in the access request, and send the connection information and the access request to the access queue. When returning a processing result of the access request, the storage unit returns the connection information. The network device determines a client corresponding to the access request based on the connection information, and returns the processing result to the client corresponding to the access request.

[0012] In the solution provided in this application, the network device simultaneously sends connection information and access request to the access queue, in order to accurately distinguish multiple clients, and returns connection information when the storage unit returns processing result, and determines the client corresponding to the access request according to the returned connection information. Therefore, when multiple clients simultaneously serve the access queue in large-scale networking connection, multiple clients can be accurately distinguished and processing result can be returned, and application scenarios can be effectively expanded.

[0013] Referring to the first aspect, in one possible implementation of the first aspect, the information about the multiple clients is connection information generated when the multiple clients respectively establish connections to the network device. When receiving an access request from any one of the multiple clients, the network device is configured to allocate a local identifier to a client identifier carried in the access request, where the local identifier uniquely identifies the client, and establish a correspondence relationship between the client identifier, the local identifier, and the connection information corresponding to the client, and replace the client identifier carried in the access request with the local identifier; when receiving a processing result of the access request returned by the storage unit, the network device obtains the local identifier from the processing result, and determines connection information corresponding to the client based on the local identifier, and returns the processing result to the client corresponding to the connection information.

[0014] It should be understood that the client identifier is set by the client, and the client identifiers set by different clients may be the same. Therefore, the clients cannot be accurately distinguished based on the client identifier. The local identifier is obtained by the network device converting the client identifier of each client, and is unique. Each client corresponds to a different local identifier. Therefore, the clients can be accurately distinguished based on the local identifier.

[0015] In the solution provided in the present application, the network device allocates a local identifier to the client identifier in the access request to uniquely identify the client, and then establishes a correspondence between the client identifier, the local identifier, and the connection information corresponding to the client, to avoid the situation where the client identifiers set by different clients are the same and different clients cannot be distinguished. In this way, multiple clients can be accurately distinguished, and after the storage unit returns the processing result, the connection information corresponding to the client is determined based on the local identifier. Therefore, when multiple clients simultaneously serve the access queue in a large-scale networking connection, multiple clients can be accurately distinguished and the processing result can be returned, and the application scenario can be effectively expanded.

[0016] Referring to the first aspect, in one possible implementation of the first aspect, a remote direct memory access (RDMA) connection is established between each of a plurality of clients and a network device, and the connection information is a queue pair (QP) generated when the RDMA connection is established.

[0017] According to a second aspect, the present application provides a data access method. The method includes the steps of: a network device receiving access requests sent by a plurality of clients connected to the network device; and sending the access requests to an access queue of a storage unit. The network device receives a processing result of the access requests in the access queue, which is returned by the storage unit after the storage unit executes the access requests. The network device returns the processing result of the access requests returned by the storage unit to a client corresponding to the access requests.

[0018] Referring to the second aspect, in one possible implementation of the second aspect, the network device stores a correspondence relationship between information about a plurality of clients and access queues, and the network device sends access requests of the plurality of clients to the access queues of the storage unit according to the mapping relationship.

[0019] Referring to the second aspect, in one possible implementation of the second aspect, the information regarding the multiple clients is connection information generated when the multiple clients each establish a connection to the network device, and the step of the network device sending access requests of the multiple clients to an access queue of the storage unit includes a step of determining an access queue based on the connection information and correspondence corresponding to the client involved in the access request when an access request of any one of the multiple clients is received, and a step of sending the connection information and the access request to the access queue. Storage The processing result returned by the unit includes connection information, and the step of the network device returning the processing result of the access request returned by the storage unit to the client corresponding to the access request includes a step of the network device determining the client corresponding to the access request based on the connection information, and a step of returning the processing result to the client corresponding to the access request.

[0020] Referring to the second aspect, in one possible implementation of the second aspect, the information about the multiple clients is connection information generated when the multiple clients establish connections to the network device, and the step of the network device sending the access requests of the multiple clients to the access queue of the storage unit includes the steps of: when the access request of any one of the multiple clients is received, assigning a local identifier to a client identifier carried in the access request, where the local identifier uniquely identifies the client; establishing a correspondence between the client identifier, the local identifier, and the connection information corresponding to the client; replacing the client identifier carried in the access request with the local identifier; and sending the access request to the access queue corresponding to the connection information. The step of the network device returning the processing result of the access request returned by the storage unit to the client corresponding to the access request includes the steps of: when the network device receives the processing result of the access request returned by the storage unit, obtaining a local identifier from the processing result; determining connection information corresponding to the client based on the local identifier; and returning the processing result to the client corresponding to the connection information.

[0021] Referring to the second aspect, in one possible implementation of the second aspect, a remote direct memory access (RDMA) connection is established between each of the multiple clients and the network device, and the connection information is a queue pair (QP) generated when the RDMA connection is established.

[0022] According to a third aspect, the present application provides a network device including a receiving unit and a storage unit. The receiving unit is configured to receive access requests sent by a plurality of clients connected to the network device, and the sending unit is configured to send the access requests to an access queue of the storage unit. The receiving unit is further configured to receive a processing result of the access requests in the access queue returned by the storage unit after the storage unit executes the access requests, and the sending unit is further configured to return the processing result of the access requests returned by the storage unit to a client corresponding to the access requests.

[0023] With reference to the third aspect, in one possible implementation of the third aspect, the network device further includes a storage unit, the storage unit is configured to store a correspondence relationship between information about the multiple clients and the access queues, and the sending unit is specifically configured to send access requests of the multiple clients to the access queues of the storage unit according to the mapping relationship.

[0024] Referring to the third aspect, in one possible implementation of the third aspect, the information about the multiple clients is connection information generated when the multiple clients respectively establish connections to the network device. The network device further includes a processing unit. The processing unit is configured to determine an access queue based on the connection information and the correspondence relationship corresponding to the client carried in the access request when receiving an access request of any one of the multiple clients. The sending unit is specifically configured to send the connection information and the access request to the access queue. The sending unit is further configured to determine a client corresponding to the access request based on the connection information, and return a processing result to the client corresponding to the access request.

[0025] Referring to the third aspect, in one possible implementation of the third aspect, the information about the multiple clients is connection information generated when the multiple clients respectively establish connections to the network device, and the network device further includes a processing unit. The processing unit is configured to allocate a local identifier to a client identifier carried in the access request when receiving an access request from any one of the multiple clients, so that the local identifier uniquely identifies the client, and is configured to establish a correspondence between the client identifier, the local identifier, and the connection information corresponding to the client, and replace the client identifier carried in the access request with the local identifier. The sending unit is specifically configured to send the access request to an access queue corresponding to the connection information. The processing unit is further configured to obtain a local identifier from the processing result when receiving a processing result of the access request returned by the storage unit, and to determine connection information corresponding to the client based on the local identifier. The sending unit is further configured to determine a client corresponding to the access request based on the connection information, and to return the processing result to the client corresponding to the access request.

[0026] Referring to the third aspect, in one possible implementation of the third aspect, a remote direct memory access (RDMA) connection is established between each of the multiple clients and the network device, and the connection information is a queue pair (QP) generated when the RDMA connection is established.

[0027] According to a fourth aspect, the present application provides a computing device, the computing device including a processor and a memory, the processor and the memory being connected using an internal bus, the memory storing instructions, and the processor calling the instructions in the memory to execute the data access method provided in any one of the second aspect and the implementations of the second aspect.

[0028] According to a fifth aspect, the present application provides a computer storage medium. The computer storage medium stores a computer program. When the computer program is executed by a processor, the steps of the data access method of the second aspect and any one of the implementations of the second aspect can be performed.

[0029] According to a sixth aspect, the present application provides a computer program product. The computer program comprises instructions. When the computer program is executed by a computer, the computer is enabled to perform steps of a data access method provided in any one of the second aspect or implementations of the second aspect.

[0030] In order to more clearly describe the technical solutions of the embodiments of the present invention, the following briefly introduces the accompanying drawings used in describing the embodiments. It is clear that the accompanying drawings in the following description illustrate some embodiments of the present invention, and those skilled in the art can further derive other drawings from these accompanying drawings without creative efforts. [Brief description of the drawings]

[0031] [Figure 1] FIG. 2 is a schematic diagram of writing data to a solid-state drive according to an embodiment of the present application. [Diagram 2] FIG. 2 is another schematic diagram of writing data to a solid-state drive according to an embodiment of the present application. [Diagram 3] FIG. 1 is a schematic diagram of a system architecture according to an embodiment of the present application. [Figure 4] 1 is a schematic flowchart of a connection establishment method according to an embodiment of the present application; [Diagram 5] 2 is a schematic flowchart of a data writing method according to an embodiment of the present application; [Figure 6] 2 is a schematic diagram of a submission queue description structure format according to an embodiment of the present application; [Figure 7] 2 is a schematic flowchart of a data reading method according to an embodiment of the present application; [Figure 8] 4 is a schematic flowchart of another data writing method according to an embodiment of the present application. [Figure 9] 4 is a schematic flowchart of another data reading method according to an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of the structure of a network device according to an embodiment of the present application; [Figure 11] 1 is a schematic diagram of the structure of a computing device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The following clearly and completely describes the technical solutions of the embodiments of the present application with reference to the accompanying drawings. It is obvious that the described embodiments are only some rather than all embodiments of the present application.

[0033] To aid those skilled in the art in better understanding, some terms and related art in the present application will first be explained with reference to the accompanying drawings.

[0034] A host, also referred to as a client, may specifically include a physical machine, a virtual machine, a container, etc. A host is configured to produce or consume data and may be, for example, an application server, a distributed file system server, etc.

[0035] The host's network devices are devices used by the host for data communication, and may specifically include a network interface controller (NIC) or an RNIC.

[0036] The access request of the host mainly includes a data read / write operation, that is, the host writes generated data to a storage unit of the storage device, or reads data from a storage unit of the storage device.

[0037] Storage devices, also called servers, may specifically include devices capable of storing data and taking the form of external centralized storage or distributed storage, for example storage servers or distributed database servers.

[0038] The network device of the storage device is a device used by the storage device for data communication, and specifically may include a NIC, an RNIC, etc. The storage unit of the storage device is a device used by the storage device for persistent data storage, and is, for example, an SSD.

[0039] Submission queue (SQ) and doorbell of solid-state storage (SSD): In a storage device, the CPU of the storage device and the SSD communicate according to the NVMe protocol. In the initialization phase when the storage device is started, the CPU of the storage device establishes a submission queue (SQ) and a completion queue (CQ) for the SSD in the memory of the storage device according to the NVMe protocol, and creates a doorbell in the SSD. The CPU saves a command sent to the SSD in the SQ, writes the position of the command in the SQ to the doorbell, and gets the command from the SQ for execution using the SSD. After executing the command, the SSD saves information of the executed command in the completion queue. The CPU can determine the executed command by reading the information of the executed command in the completion queue, and delete the executed command from the transmission queue.

[0040] The RDMA communication protocol is a set of protocol specifications followed by computing devices configured to perform RDMA operations. Currently, there are three RDMA-supported communication protocols: infiniBand (IB) protocol, RDMA over converged ethernet (RoCE) protocol, and internet wide area RDMA (IWARP) protocol. All three protocols can be used using the same set of APIs, but the physical and link layers are different in the three protocols. When a host device communicates with a storage device through RDMA, it can create a send queue (SQ) in the host's network interface card, and correspondingly, a receive queue (RQ) corresponding to the send queue can be created in the storage device's network interface card. The send queue and the receive queue form a queue pair (QP). The address of the queue is mapped to the application's virtual address, so the application can use the QP to directly send data to the storage device's network interface card, and then the data can be stored in the storage device's memory.

[0041] Currently, when sending data by RDMA, the host first sends data to the memory of the storage device, and then uses the CPU of the storage device to move the data from the memory to the SSD. Figure 1 is a schematic diagram of a data writing scenario. The network device 1110 of the host 110 first performs an RDMA operation to write data to the network device (i.e., the RNIC 1240) of the storage device 120, and then the RNIC 1240 writes the data to the memory 1220 with the help of the RNIC driver 1211 in the CPU 1210. The storage software 1212 in the storage device 120 recognizes that the data has been written to the memory 1220 in an event or interrupt manner. Then, the CPU 1210 uses the SSD driver 1213 to control the SSD 1230 to move the data from the memory 1220 to the SSD 1230 for persistent storage. After completing persistent data storage, the SSD 1230 notifies the storage software 1212 in an interrupt manner. Finally, the CPU 1210 returns a write completion notification message to the host 110 through the RNIC 1240 .

[0042] When storing data persistently, the participation of the CPU (including the RNIC driver, storage software, and SSD driver) is required to complete the entire storage process, which consumes a large amount of CPU resources.

[0043] To reduce CPU usage and processing latency, data can be written directly to SSD, and the SQ address of SSD and the QP between the host and the storage device can be combined one by one. Figure 2 is a schematic diagram of another data writing scenario. Application server 210, application server 220, and application server 230 are connected to storage server 240 through an RDMA network. The configuration of each application server is similar. Application server 210 is used as an example. Application server 210 includes CPU 211 and memory 212, and is connected to RNIC 213. Storage server 240 includes CPU 241 and memory 242, and is connected to RNIC 243 and a persistent storage medium. Here, SSD 244 is used as an example for explanation. It should be understood that the persistent storage medium includes but is not limited to SSD 244. Each application server is connected to a storage server to form a QP. Therefore, the memory 242 of the storage server 240 has multiple QPs, for example, QP 1, QP 2, and QP 3, and each QP corresponds to the connection between the storage server 240 and one application server. In addition, the SSD 244 includes multiple SQs and CQs, and the maximum number of SQs and the maximum number of CQs can be supported up to 64,000. However, the maximum number of 256 is usually selected at present in consideration of memory and performance. To explain the specific binding process of SQs and QPs, the application server 210 is used as an example. The application server 210 registers the memory 212 required for data communication with the RNIC 213, and the storage server 240 registers the memory 242 required for data communication with the RNIC 243, so that the RNIC 213 and the RNIC 243 can operate the memory 242 and the memory 212 in the RDMA manner. In addition, the storage server 240 binds the SQs in the SSD 244 to the stored QPs one by one, for example, binds the QP 1 connected to the application server 210 to the SQ 1. Then, the address of the SQ 1 is mapped, and the virtual address obtained by the mapping is registered in the RNIC 243.RNIC 243 sends the address of SQ 1 to RNIC 213 through RDMA connection, so that RNIC 213 can directly and remotely operate the address of SQ 1 in SSD 244. When writing data, application server 210 uses CPU 211 to generate data, stores the generated data in memory 212, and then writes the data to memory 242 of storage server 240 through RNIC 213, and notifies SSD 244 to move the data in memory 242 to SSD 244 for persistent storage based on the SQ address of SSD 244. If another application server needs to write data to storage server 240, the process is similar to the above description, and the details will not be described again here.

[0044] Note that in this solution, QPs are bound to SQs in the SSD one by one, so data can be written directly to the SSD without the involvement of the storage server's CPU and software. However, this is limited by the number of SQs in the SSD. If the number of connections is too large, this solution is no longer applicable, i.e. it cannot support large-scale networking scenarios.

[0045] Based on this description, the present application provides a data access method. When the number of connections to a storage device is far greater than the number of SQs supported by SSD, the submission queue description structure format (SQE) of SSD is extended, or the client identifier of application server is converted, so that the storage device can send the access requests of multiple clients to the access queue of storage unit. In other words, multiple connections of a storage device can be combined into one SQ to support large-scale networking connections and expand applicable scenarios.

[0046] The technical solutions of the embodiments of the present application are applicable to systems that need to access persistent storage media remotely, such as in distributed storage and high performance computing (HPC), especially in large-scale networking scenarios with a large number of connections. For example, in distributed storage, a storage device is connected to many application servers simultaneously. When the storage device needs to support each application server in directly accessing the SSD, the data access method provided in the embodiments of the present application can be used for the distributed storage system, so that the bandwidth bottleneck that exists during data reading / writing can be resolved and the data reading / writing efficiency can be improved.

[0047] 3 is a schematic diagram of a system architecture according to an embodiment of the present application. As shown in FIG. 3, the system 300 includes an application server 310, an application server 320, an application server 330, and a storage server 340. The application server 310, the application server 320, and the application server 330 communicate with the storage server 340 through an RDMA network. 34 0. Application server 310 includes a CPU 311 and a memory 312, and is connected to RNIC 213. The configurations of application server 320 and application server 330 are similar to that of application server 310. Storage server 340 includes a CPU 341 and a memory 342, and is connected to RNIC 343 and a storage unit. Here, SSD 344 is used as an example for explanation. It should be understood that the storage unit may be, but is not limited to, SSD 344. Since application server 310, application server 320, and application server 330 are all connected to storage server 340, there are three QPs in memory 342 of storage server 340, namely QP 1, QP 2, and QP 3. Storage server 340 binds QP 1 and QP 2 to SQ 1, and binds QP 3 to SQ 2. After the procedure is completed, further data read / write operations can be performed. For example, application server 310 writes data to storage server 340. Application server 310 generates data using CPU 311, stores the generated data in memory 312, and then uses RNIC 313 to write the data to memory of storage server 340. 3 The application server 310 writes data and data description information to the memory 242. The data description information includes the start address of the data in the memory 242, the data length, the operation type, etc. To comply with and be consistent with the NVMe protocol, the size of the data description information is preferably 64 bytes. After the storage server 340 receives the data written by the application server 310, the RNIC 343 determines, based on the preset binding relationship, that the SQ corresponding to the QP 1 generated when the application server 310 is connected to the storage server 340 is SQ 1. 1 0 fills in a submission queue description structure format (SQE) field corresponding to SQ 1 based on the data description information, and uses a reserved field in the SQE to store a QP number (QPN), i.e., a number corresponding to QP 1. Alternatively, the storage server 340 converts the client identifier carried in the data description information into a local identifier, stores the local identifier in the SQE, and then notifies the SSD 344 to move the data in the memory 342 to the SSD 344 for persistent storage. After the storage is completed, the SSD 344 copies the QPN information to the CQE of the CQ, and the storage server 340 determines the corresponding QPN based on the CQE. Alternatively, the storage server 340 determines the corresponding client identifier and the corresponding QPN based on the local identifier in the CQE, and then finds the corresponding QP based on the QPN, and returns a data write completion message to the application server 310 using the QP.

[0048] In this embodiment of the present application, RNIC 313, RNIC 323, and RNI C 333 may be a programmable RNIC. SSD 344 is a programmable SSD and can actively recognize and report the completion status of SQ. Application server 310, application server 320, application server 330, and storage server 340 may include forms such as physical machines, virtual machines, and containers, and may be deployed on one or more computing devices (e.g., a central server) in a cloud environment, or one or more computing devices (e.g., a server) in an edge environment.

[0049] Compared with the data access system shown in Fig. 2, it can be seen that in the data access system shown in Fig. 3, the storage server can simultaneously bind connections (QP) to multiple application servers to the same SQ and send access requests of multiple application servers to that SQ. This breaks the inherent quantity limit of SQs in SSDs, can support large-scale networking connections, and expands the application scenarios.

[0050] With reference to the schematic diagram of the system architecture shown in Fig. 3, the following describes the data access method provided in the embodiment of the present application with reference to Fig. 4. First, the process of connection establishment and memory registration before data access is described. An example of application server 310 establishing a connection to storage server 340 is used for the description. The process of connection establishment and memory registration of other application servers is similar to that of application server 310. As shown in Fig. 4, the procedure includes the following steps:

[0051] S401: The application server 310 establishes an RDMA connection to the storage server 340.

[0052] Optionally, the application server 310 may establish an RDMA connection to the storage server 340 according to any one of the IB, RoCE, or IWARP protocols.

[0053] Specifically, the application server 310 and the storage server 340 register memory addresses (which may be contiguous virtual memory or contiguous physical memory space) required for data communication, and provide the memory addresses to the network device as a virtual contiguous buffer. This buffer uses virtual addresses. For ease of understanding and description, in this embodiment of the present application, an example in which the network device is an RNIC is used for description, and no further distinction is made in the following description. For example, the application server 310 registers memory 312 with the RNIC 313, and the storage server 340 registers memory 342 with the RNIC 343. It should be understood that during registration, the operating systems of the application server 310 and the storage server 340 check the permission of the registered block. The registration process writes a mapping table between the virtual address and the physical address of the memory that needs to be registered to the RNIC. In addition, during memory registration, the permission of the corresponding memory area is set, and the permission includes local write, remote read, remote write, etc. After registration, the memory registration process locks the memory page. To prevent memory pages from being replaced, the registration process also needs to maintain the mapping between physical and virtual memory.

[0054] Optionally, when performing memory registration, the application server 310 and the storage server 340 may register for all memories of the application server 310 and the storage server 340, or may register for some randomly selected memories. During registration, the starting address and data length of the memory that needs to be registered are provided to the RNIC, so that the RNIC can determine the memory that needs to be registered.

[0055] Note that each memory registration generates a remote identifier (key) and a local identifier accordingly. The remote identifier is used by the remote host to access the local memory, and the local identifier is used by the local host to access the local memory. For example, during a data receive operation, the storage server 340 provides the application server 340 with the remote identifier generated by the memory registration, so that the application server 310 can remotely access the system memory 342 of the storage server 310 during an RDMA operation. In addition, the same memory buffer may be registered multiple times (configured with different operation permissions), and a different identifier is generated for each registration.

[0056] In addition, in the process of establishing an RDMA connection, the application server and the storage server negotiate to create a QP. When a QP is created, an associated send queue SQ and an associated receive queue RQ are created. After the creation is complete, the application server 310 can communicate with the storage server 340 using the QP.

[0057] It should be understood that after the application server 310 establishes an RDMA connection to the storage server 340, the application server 310 can remotely access the memory 342 of the storage server 340 in an RDMA manner.

[0058] S402: The storage server 340 maps the SQ address and the doorbell address of the SSD 344, and registers the address obtained by the mapping in the RNIC 343.

[0059] Specifically, in the initialization phase of storage server 340, storage server 340 establishes an SQ for SSD 344 in memory 342 and establishes a doorbell in SSD 344 to implement communication between CPU 341 in storage server 340 and SSD 344. It should be noted that the SQ address and doorbell address are addresses in the kernel mode memory address space, and cannot be directly registered in RNIC 343, but can only be registered after being translated into a user mode virtual address.

[0060] In addition, the storage server 340 maps the SQ addresses and doorbell addresses of the SSDs to logically consecutive user mode virtual addresses, and then provides the virtual addresses obtained by the mapping to the storage server's RNIC 343 for registration. The registration process is similar to the memory registration process, and the details are not described again here. Optionally, the storage server 340 may complete the mapping process in a memory mapping (MMAP) manner to map the SQ addresses and doorbell addresses to user mode virtual addresses, in order to ensure normal communication between the SSDs and the RNIC 343.

[0061] S403: The storage server 340 binds the QP to the SQ of the SSD 344.

[0062] Specifically, during the initialization phase, multiple SQ addresses are assigned to SSD 344. When establishing an RDMA connection, RNIC 343 of storage server 340 and the RNICs of multiple application servers, including application server 310, also create multiple QPs. Management software in storage server 340 binds the SQ addresses to QPs and sends the bindings to RNIC 343 for storage.

[0063] It should be noted that for the connection between each application server and the storage server 340, the storage server 340 can accurately distinguish the application server by a method such as numbering. In other words, for each QP, there is a unique QP number (QPN) corresponding to the QP.

[0064] In addition, the storage server 340 combines N QPs into one SQ to support large-scale networking connection. The specific value of N can be set based on actual requirements, for example, can be set to 100. This is not limited in this application.

[0065] It can be seen that after the storage server 340 binds the SQ addresses to QPs, the storage server 340 can identify the QP corresponding to each SQ address based on the stored binding relationship to distinguish different clients or application servers.

[0066] It will be understood that the method steps shown in FIG. 4 are executed, the application server 310 and the storage server 340 can successfully establish an RDMA connection to perform data transmission, the application server 310 can remotely operate the memory 342 of the storage server 340, and the storage server 340 can persistently store the data written by the application server 310 based on the binding relationship between QP and SQ.

[0067] With reference to the system architecture shown in Fig. 3 and the connection establishment method procedure shown in Fig. 4, the data writing procedure is described in detail below. Take an example in which the application server 310 writes data to the storage server 340. As shown in Fig. 5, this procedure includes the following steps:

[0068] S501: The application in the application server 310 writes the data to be written to the local memory.

[0069] Specifically, after an application in the application server 310 generates data that needs to be written to the SSD 344 of the storage server 340 , the application first saves the data in the memory 312 of the application server 310 .

[0070] S 502 : The RNIC 313 of the application server 310 writes the write target data and the write target data description information to the memory 342 of the storage server 340 .

[0071] Specifically, an application in the application server 310 sends an RDMA request to the RNIC 313 of the application server 310, where the request includes the address of the data to be written in the memory 312 (e.g., including a start address and a data length). The RNIC 313 then extracts the data to be written from the memory 312 of the application server 310 based on the request, and encapsulates the address of the data to be written in the storage server 340 (including a start address and a data length) and a remote identifier sent by the storage server 340 to be used to operate the memory corresponding to the address in a dedicated packet. In addition, information describing the data to be written is also encapsulated in the dedicated packet, where the information describing the data to be written includes the start address and the data length of the data to be written in the storage server 340, a data operation type (i.e., a data write operation), etc. The dedicated packet is then sent to the RNIC 343 of the storage server 340 using a QP. After receiving the dedicated packet, the RNIC 343 of the storage server 340 determines whether the application server 310 has the authority to operate the memory 342 of the storage server 340 based on the remote identifier in the packet, and after determining that the application server 310 has the authority, writes the data to be written to the memory corresponding to the address in the packet, and also writes description information of the data to be written to the memory 342.

[0072] S503: The RNIC 343 of the storage server 340 fills in an SQE corresponding to an SQ based on a QP corresponding to the data to be written and the write data description information.

[0073] Specifically, after the application server 310 uses the QP to write the data to be written and the information describing the data to be written to the memory 342 of the storage server 340, the RNIC 343 of the storage server 340 can determine the SQ corresponding to the QP based on the pre-stored binding relationship. Each SQ includes one or more SQEs. The format of each SQE complies with the NVMe protocol specification, and the size of each SQE is 64 bytes. FIG. 6 is a schematic diagram of the SQE format. The SQE includes a specific command field, a pending field, an SQ identifier field, an SQ head pointer field, a status field, a command identifier field, etc. The RNIC 343 of the storage server 340 fills the SQE corresponding to the SQ based on the information describing the data to be written.

[0074] Note that when filling the SQE, the RNIC 343 of the storage server 340 extends a reserved field in the SQE and uses this reserved field to store the QPN corresponding to the QP, so that the SQE carries the QPN information.

[0075] S504: The RNIC 343 of the storage server 340 writes write data notification information to the doorbell address of the SSD 344.

[0076] Specifically, the RNIC 343 of the storage server 340 writes write data notification information to the doorbell address of the SSD 344, where the write data notification information includes the SQ address to which the SQE was written, and the write data notification information notifies the SSD 344 to read the SQE in this SQ address.

[0077] S505: The SSD 344 reads the SQE in the SQ address based on the write data notification information in the doorbell address, and moves the data to be written from the memory 342 of the storage server 340 to the SSD 344 based on the contents of the SQE.

[0078] Specifically, after receiving the write data notification information written to the doorbell address, the SSD 344 is awakened, and then reads the SQE in the SQ address included in the write data notification information to determine that the operation is a data write operation. Then, according to the address carried in the SQE, the write target data is found in the memory 342 of the storage server 340, and the write target data is moved to the SSD 344 to complete the persistent storage.

[0079] It can be seen that the data to be written can be moved from the memory 342 of the storage server 340 to the SSD 344 without any software or CPU involvement. This process is directly accomplished by the SSD 344. This reduces the CPU usage of the storage server 340 and effectively reduces costs.

[0080] S506: After completing the persistent data save, the SSD 344 copies the QPN information in the SQE to the CQE of the CQ and notifies the RNIC 343 that the write command has been completed.

[0081] Specifically, in NVMe, each SQ corresponds to one CQ, each CQ includes one or more CQEs, and the size of each CQE is also 64 bytes. The format of each CQE is similar to the format of the SQE shown in Figure 6. After completing the persistent data save, the SSD 344 copies the QPN field in the SQE to the pending field in the CQE, and then notifies the RNIC 343 that the write command is completed.

[0082] S507: The RNIC 343 of the storage server 340 determines the QP corresponding to the QPN information based on the QPN information in the CQE, and notifies the application server 310 that the data writing has been completed using the QP.

[0083] Specifically, after receiving a write command completion notification sent by SSD 344, RNIC 343 reads the CQE from the CQ to obtain QPN information, determines the QP corresponding to the QPN information based on the QPN information, and then notifies the application server 310 using the QP that the data writing has been completed, thereby completing the entire data writing procedure.

[0084] It can be seen that in the process of writing the data to be written to the SSD 344, if there are multiple QPs, the multiple QPs are combined into one SQ and the reserved field in the SQE is used to store the QPN. After the data writing is completed, the QPN in the CQE can be used to accurately find the corresponding QP to respond with a completion message. This can effectively support large-scale networking connections and expand the applicable scenarios.

[0085] The method steps illustrated in Figure 5 detail the process of writing data from the application server to the SSD. Correspondingly, the application server can also read data from the SSD. The following details the data reading procedure. As shown in Figure 7, the procedure includes the following steps:

[0086] S701: The RNIC 313 of the application server 310 writes the read target data description information to the memory 342 of the storage server 340.

[0087] Specifically, an application in the application server 310 generates a data read request and then sends the data read request to the RNIC 313 of the application server 310, where the read request includes the address of the data to be read in the SSD 344 (including the starting address and data length) and the address of the data to be stored in the memory 342 of the storage server 340 after the data is read from the SSD 344.

[0088] Furthermore, the RNIC 313 of the application server 310 uses the stored remote identifier to operate the memory 342 of the storage server 340, and writes the read target data description information to the memory 342 of the storage server 340. The read target data description information includes the start address and address length of the read target data in the SSD 344, the address in the memory 342 of the storage server 340 where the read target data needs to be saved, and the data operation type (i.e., data read operation). The RNIC 343 fills in the SQE corresponding to the SQ based on the QP corresponding to the write target data and the write target data description information.

[0089] S702: The RNIC 343 of the storage server 340 fills in an SQE corresponding to an SQ based on a QP corresponding to the data to be read and read target data description information.

[0090] Specifically, after the application server 310 writes the read target data description information to the memory 342 of the storage server 340 using the QP, the RNIC 343 of the storage server 340 can determine the SQ corresponding to the QP based on the pre-stored binding relationship, and the RNIC 343 of the storage server 340 fills in the SQE corresponding to the SQ based on the read target data description information.

[0091] Similarly, when filling the SQE, the RNIC 343 of the storage server 340 extends a reserved field in the SQE and uses the reserved field to store the QPN corresponding to the QP, so that the SQE carries the QPN information.

[0092] S703: The RNIC 343 of the storage server 340 writes read data notification information to the doorbell address of the SSD 344.

[0093] Specifically, the RNIC 343 of the storage server 340 writes read data notification information to the doorbell address of the SSD 344, where the read data notification information includes the SQ address to which the SQE was written, and the read data notification information notifies the SSD 344 to read the SQE in this SQ address.

[0094] S704: The SSD 344 reads the SQE in the SQ address based on the read data notification information in the doorbell address, and moves the data to be read from the SSD 344 to the memory 342 of the storage server 340 based on the contents of the SQE.

[0095] Specifically, after receiving the read data notification information written to the doorbell address, the SSD 344 is awakened, and then reads the SQE in the SQ address included in the read data notification information and determines that the operation is a data read operation. Then, data is extracted from the SSD 344 according to the address carried in the SQE, and the data is moved to the memory 342 of the storage server 340.

[0096] S705: After completing the data movement, the SSD 344 copies the QPN information in the SQE to the CQE of the CQ, and notifies the RNIC 343 that the read command has been completed.

[0097] Specifically, after SSD 344 moves the data to memory 342 of storage server 340, it copies the QPN field in the SQE to the pending field in the CQE and then notifies RNIC 343 that the read command is complete.

[0098] S706: The RNIC 343 of the storage server 340 determines the QP corresponding to the QPN information based on the QPN information in the CQE, writes the data to be read to the memory 312 of the application server 310 using the QP, and notifies the application server 310 that the data reading is complete.

[0099] Specifically, after receiving a read command completion notification sent by SSD 344, RNIC 343 reads the CQE from the CQ to obtain QPN information, determines the QP corresponding to the QPN based on the QPN information, writes the data to be read to memory 312 of application server 310 using the QP, and then notifies application server 310 that the data reading is completed, thereby completing the entire data reading procedure.

[0100] It should be noted that the method embodiment shown in Fig. 7 and the method embodiment shown in Fig. 5 are based on the same idea, and can be mutually referenced in specific implementation processes, and for the sake of brevity, details will not be described again here.

[0101] With reference to the system architecture shown in Figure 3 and the connection establishment method procedure shown in Figure 4, another data writing method is described in detail below. As shown in Figure 8, this procedure includes the following steps:

[0102] S801: The storage server 340 receives the data to be written and the description information of the data to be written that have been written to the memory 342 by the application server.

[0103] Specifically, each application server connected to storage server 340 writes data and data description information generated by an application to memory 342 of storage server 340 using the QP of each application server through the RNIC. For example, application server 310 uses QP 1 to write data to be written and description information of data to be written to memory 342 of storage server 340, and application server 320 uses QP 2 to write data to be written and description information of data to be written to memory 342 of storage server 340. The description information of data to be written includes a start address and data length of data to be written in storage server 340, a data operation type (i.e., a data write operation), etc.

[0104] Please note that the write target data description information further carries a client identifier (i.e., cid). The client identifier is set by each application server. Therefore, the client identifiers set by different application servers may be the same. For example, the client identifier set by application server 310 is cid1, and the client identifier set by application server 320 is also cid1.

[0105] It should be understood that for each application server, there is a correspondence between the connection (QP) between the application server and the storage server 340 and the client identifier set by the application server, i.e., the QPN corresponding to the application server can be determined based on the client identifier.

[0106] S802: The RNIC 343 of the storage server 340 converts the client identifier of each application server into a local identifier, and establishes a mapping table between the client identifier and the local identifier.

[0107] Specifically, because the client identifier is set by the application server, the client identifiers set by different application servers may be the same. Therefore, different application servers cannot be accurately distinguished based on the client identifier. Therefore, RNIC 343 of storage server 340 needs to convert the client identifier of each application server into a locally unique identifier so that different application servers can be accurately distinguished.

[0108] For example, the client identifier carried in the information describing data to be written written by application server 310 to memory 342 is 00000001, the client identifier carried in the information describing data to be written written by application server 320 to memory 342 is also 00000001, and the client identifier carried in the information describing data to be written written by application server 310 to memory 342 is 00000101, and RNIC 343 performs a conversion on the received client identifiers corresponding to the application servers and converts the received client identifiers corresponding to the application servers into locally unique identifiers. For example, RNIC 343 converts the client identifier corresponding to application server 310 to 00000001, the client identifier corresponding to application server 320 to 00000010, and the client identifier corresponding to application server 330 to 00000011. It will be appreciated that after conversion, the identification information corresponding to each application server is unique, and different application servers can be accurately distinguished by using the converted local identifiers.

[0109] In addition, after completing the identifier conversion, RNIC 343 further establishes a mapping table between client identifiers and local identifiers. Optionally, RNIC 343 may use a hash table to record the mapping relationship between client identifiers and local identifiers. In this hash table, the key of an application server is the local identifier, and the value of an application server is the client identifier and the corresponding QPN. RNIC 343 can use the hash table to query the client identifier and the corresponding local identifier of each application server.

[0110] S803: The RNIC 343 of the storage server 340 fills in an SQE corresponding to the SQ based on the write target data description information.

[0111] Specifically, after the storage server 340 receives through the RNIC that each application server writes application-generated data and data description information to the memory 342 of the storage server 340 according to the respective QP, the RNIC 343 can determine the SQ corresponding to each QP according to the pre-stored binding relationship, and then fill the SQE corresponding to the SQ according to the write target data description information. It should be noted that in the process of filling the SQE, the RNIC 343 modifies the identifier field in the SQE and fills the local identifier corresponding to the application server in this field. For example, for the application server 320, the RNIC 343 fills the field with 00000010 instead of 00000001.

[0112] S804: The RNIC 343 of the storage server 340 writes write data notification information to the doorbell address of the SSD 344.

[0113] Specifically, the RNIC 343 writes write data notification information to the doorbell address of the SSD 344, where the write data notification information includes the SQ address to which the SQE was written, and the write data notification information notifies the SSD 344 to read the SQE in this SQ address.

[0114] S805: The SSD 344 reads the SQE in the SQ address based on the write data notification information in the doorbell address, and moves the data to be written from the memory 342 of the storage server 340 to the SSD 344 based on the contents of the SQE.

[0115] Specifically, after receiving the write data notification information written to the doorbell address, the SSD 344 is awakened, and then reads the SQE in the SQ address included in the write data notification information to determine that the operation is a data write operation. Then, according to the address carried in the SQE, the write target data is found in the memory 342 of the storage server 340, and the write target data is moved to the SSD 344 to complete the persistent storage.

[0116] S806: After completing the persistent data storage, the SSD 344 notifies the RNIC 343 that the write command is completed.

[0117] Specifically, after SSD 344 completes the persistent data save, it fills a CQE in the CQ corresponding to the SQ. The format of the CQE is consistent with that of the SQE, and the CQE also includes an identifier field, which stores a local identifier corresponding to the application server, and then notifies RNIC 343 that the write command is completed.

[0118] S807: The RNIC 343 of the storage server 340 queries a mapping table of client identifiers and local identifiers based on the local identifier in the CQE, determines the client identifier corresponding to the local identifier, determines the QPN corresponding to the client identifier, and notifies the application server that the data writing is complete using the QP corresponding to the QPN.

[0119] Specifically, after receiving a write command completion notification sent by SSD 344, RNIC 343 reads the CQE from the CQ to obtain a local identifier, queries a mapping table of client identifiers and local identifiers based on the local identifier to obtain a client identifier and QPN corresponding to the local identifier, then determines a corresponding QP based on the QPN, and finally uses the QP to notify the application server that the data writing is completed, thereby completing the entire data writing procedure.

[0120] It can be seen that in the process of writing the target data to the SSD 334, when there are multiple application servers (i.e., multiple QPs) and the client identifiers set by the application servers may be the same, the multiple QPs are combined into one SQ, and the client identifiers corresponding to the QPs are converted into local unique identifiers. The converted local identifiers are stored in the identifier field of the SQE. After the data writing is completed, the corresponding client identifiers and QPs can be accurately found by querying the mapping table between client identifiers and local identifiers based on the local identifiers in the CQE, and the different application servers are accurately distinguished, and a completion message is returned to the application server. This can effectively support large-scale networking connections and expand the application scenarios.

[0121] The method steps illustrated in Figure 8 detail the process of writing data from the application server to the SSD. Correspondingly, the application server can also read data from the SSD. The following details the data reading procedure. As shown in Figure 9, the procedure includes the following steps:

[0122] S901: The storage server 340 receives the read target data description information written in the memory 342 by the application server.

[0123] Specifically, the application server uses the QP of the application server to write the read target data description information to the memory 342 of the storage server 340 through the RNIC. The read target data description information includes the start address and data length of the read target data in the SSD 344, the data operation type (i.e., data read operation), etc. In addition, the read target data description information further carries a client identifier. For the specific process, please refer to the relevant description of S801, and the details will not be described again here.

[0124] S902: The RNIC 343 of the storage server 340 converts the client identifier of each application server into a local identifier, and establishes a mapping table between the client identifier and the local identifier.

[0125] Specifically, after converting the client identifier of each application server into a local unique identifier, the RNIC 343 can use a hash table to record the mapping relationship between the client identifier and the local identifier. For the specific process, please refer to the related description of S802.

[0126] S903: The RNIC 343 of the storage server 340 fills in an SQE corresponding to the SQ based on the read target data description information.

[0127] Specifically, the RNIC 343 determines the SQ corresponding to each QP according to the pre-stored binding relationship, and then fills the SQE corresponding to the SQ according to the read target data description information, and fills the converted local identifier of the application server in the identifier field of the SQE. For the specific process, please refer to the related description of S803.

[0128] S904: The RNIC 343 of the storage server 340 writes read data notification information to the doorbell address of the SSD 344.

[0129] Specifically, the RNIC 343 writes a read data notification to the doorbell address of the SSD 344, where the read data notification includes the SQ address to which the SQE was written, and the read data notification notifies the SSD 344 to read the SQE in this SQ address.

[0130] S905: The SSD 344 reads the SQE in the SQ address based on the read data notification information in the doorbell address, and moves the data to be read from the SSD 344 to the memory 342 of the storage server 340 based on the contents of the SQE.

[0131] Specifically, after receiving the read data notification information written to the doorbell address, the SSD 344 is awakened, and then reads the SQE in the SQ address included in the read data notification information, and determines that the operation is a data read operation. Then, the read target data is found from the SSD 344 according to the address carried in the SQE, and the read target data is moved to the memory 342 of the storage server 340.

[0132] S906: After completing the data movement, the SSD 344 notifies the RNIC 343 that the read command has been completed.

[0133] Specifically, after SSD 344 completes the data movement, it fills a CQE in the CQ corresponding to the SQ. The format of the CQE is consistent with that of the SQE, and the CQE also includes an identifier field, which stores a local identifier corresponding to the application server, and then notifies RNIC 343 that the read command is completed.

[0134] S907: The RNIC 343 of the storage server 340 queries a mapping table of client identifiers and local identifiers based on the local identifier in the CQE, determines a client identifier corresponding to the local identifier, determines a QPN corresponding to the client identifier, writes the data to be read to the memory of the application server using the QP corresponding to the QPN, and then notifies the application server that the data reading is completed.

[0135] Specifically, after receiving the read command completion notification sent by SSD 344, RNIC 343 reads the CQE from the CQ to obtain a local identifier, queries a mapping table of client identifiers and local identifiers based on the local identifier to obtain a client identifier and QPN corresponding to the local identifier, then determines a corresponding QP based on the QPN, and finally writes the data to be read into the memory of the application server using the QP, and notifies the application server that the data reading is completed, thereby completing the entire data reading procedure.

[0136] It should be noted that the method embodiment shown in Fig. 9 and the method embodiment shown in Fig. 5 are based on the same idea, and can be mutually referenced in specific implementation processes. For the sake of brevity, details will not be described again here.

[0137] The method of the embodiment of the present application has been described in detail above. In order to better facilitate implementing the solution of the embodiment of the present application, the following further provides correspondingly related devices that are used to cooperate in implementing the solution.

[0138] 10 is a schematic diagram of a structure of a network device according to an embodiment of the present application. As shown in FIG. 10, the network device 10 includes a receiving unit 11 and a sending unit 12.

[0139] The receiving unit 11 is configured to receive access requests sent by a plurality of clients connected to the network device 10 .

[0140] The sending unit 12 is configured to send an access request to an access queue of a storage unit.

[0141] The receiving unit 11 is further configured to receive a processing result of the access request of each of the multiple clients in the access queue, which is returned by the storage unit after the storage unit executes the access request.

[0142] The sending unit 12 is further configured to return a processing result of the access request returned by the storage unit to the client corresponding to the access request.

[0143] In one embodiment, the network device 10 further includes a storage unit 13. The storage unit 13 is configured to store a correspondence relationship between information about multiple clients and access queues. The sending unit 12 is specifically configured to send the access requests of the multiple clients to the access queues of the storage unit according to the mapping relationship.

[0144] In one embodiment, the access request includes data description information, and the network device 10 further includes a processing unit 14. The processing unit 14 is configured to fill an SQE corresponding to an access queue with the data description information, and to store each QPN information corresponding to a plurality of clients in a reserved field of the SQE.

[0145] In one embodiment, the processing unit 14 is further configured to determine a client corresponding to the processing result of the access request returned by the storage unit based on the QPN information in the CQE corresponding to the completion queue corresponding to the access queue, and the QPN information in the CQE is obtained by copying the QPN information in the SQE after the storage unit executes the access request in the access queue. The sending unit 12 is specifically configured to return the processing result to the client corresponding to the access request based on the QP corresponding to the QPN information.

[0146] In one embodiment, the access request includes data description information, where the data description information carries a client identifier. The processing unit 14 is further configured to convert the client identifier into a local identifier and establish a mapping table between the client identifier and the local identifier, where the local identifier is used to uniquely identify multiple clients.

[0147] In one embodiment, the processing unit 14 is further configured to fill the data description information in the SQE corresponding to the access queue, the SQE including a local identifier, and to query a mapping table of client identifiers and local identifiers based on the local identifier in the CQE corresponding to the completion queue corresponding to the access queue to determine a client identifier corresponding to the local identifier and a client corresponding to the processing result of the access request returned by the storage unit. The sending unit 12 is specifically configured to return the processing result to the client corresponding to the access request based on the QP corresponding to the client identifier.

[0148] It should be understood that the structure of the network device is merely an example and does not constitute a specific limitation. The units of the network device can be added, deleted, or combined as necessary. In addition, the operations and / or functions of the units in the network device are respectively used to implement the corresponding steps of the methods described in Figures 4, 5, 7, 8, and 9. For the sake of brevity, the details will not be described again here.

[0149] 11 is a schematic diagram of a structure of a computing device according to an embodiment of the present application. As shown in FIG. 11, a computing device 20 includes a processor 21, a communication interface 22, and a memory 23. The processor 21, the communication interface 22, and the memory 23 are connected to each other using an internal bus 24.

[0150] The computing device 20 may be the network device of Figure 3. The functions performed by the network device of Figure 3 are actually performed by the processor 21 of the network device.

[0151] The processor 21 may include one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0152] The bus 24 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus 24 can be classified into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line represents a bus in FIG. 11, but this does not mean that there is only one bus or only one type of bus.

[0153] The memory 23 may include volatile memory, such as random access memory (RAM). The memory 23 may alternatively include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory 23 may alternatively include a combination of the aforementioned types of memory. Program code may be used to implement the functional units illustrated in the network device 10 or to implement method steps performed by the network device in the method embodiments illustrated in Figures 4, 5, 7, 8, and 9.

[0154] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, can implement some or all of the steps recorded in any one of the above method embodiments and the functions of any of the functional units shown in FIG.

[0155] An embodiment of the present application further provides a computer program product, which, when executed on a computer or a processor, enables the computer or the processor to perform one or more steps of any one of the above-mentioned methods. When the above-mentioned units in the device are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0156] In the above embodiments, the description of the embodiments has its own focus, and for the parts not explained in detail in one embodiment, please refer to the relevant descriptions of other embodiments.

[0157] It should be understood that the sequence numbers of the above processes do not mean the execution order in various embodiments of the present application. The execution order of the processes should be determined according to the functions and internal logic of the processes, and should not be construed as limiting the implementation process of the embodiments of the present application.

[0158] It is clearly understood by those skilled in the art that the detailed working processes of the above-mentioned systems, devices and units should be referred to the corresponding processes of the above-mentioned method embodiments, and the details will not be described again here, for convenience of description.

[0159] When functions are implemented in the form of software functional units and sold or used as independent products, those functions may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application may be essentially implemented in the form of a software product, or the part that contributes to the prior art, or a part of the technical solutions. The computer software product includes some instructions stored in a storage medium, instructing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium that can store program code, such as a U-flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0160] Finally, the above embodiments are only for illustrating the technical solutions of the present application, and do not limit the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that, without departing from the scope of the technical solutions of the embodiments of the present application, modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can still be made to some technical features thereof. [Explanation of symbols]

[0161] 10 Network Devices 11 Receiving unit 12 Transmitting unit 13 Storage Unit 14 Processing Unit 20 Computing Devices 21 Processors 22 Communication Interface 23 Memory 24 Bus 110 Host 120 Storage Devices 210 Application Server 211 Central Processing Unit (CPU) 212 Memory 213 RNIC 220 Application Server 230 Application Server 240 Storage Server 241 CPU 242 Memory 244 Solid State Drive (SSD) 300 Systems 310 Application Server 311 CPU 312 Memory 313 RNIC 320 Application Server 323 RNIC 330 Application Server 333 RNI 340 Storage Server 341 CPU 342 Memory 343 RNIC 344 SSD 1110 Network Devices 1210 CPU 1211 RNIC driver 1212 Storage Software 1213 SSD Driver 1220 Memory 1230 SSD 1240 RNIC

Claims

1. A data access method for a network device, the network device configured to communicate with each of a plurality of clients by using a respective queue pair, each queue pair being formed by a transmit queue of a client and a receive queue of the network device, the method comprising: receiving, by the network device, access requests sent by the plurality of clients connected to the network device by using the respective queue pairs, and sending the access requests to an access queue of a storage unit; receiving, by the network device, a processing result of the access request in the access queue returned by the storage unit after the storage unit executes the access request; returning, by the network device, the processing result of the access request returned by the storage unit to the client corresponding to the access request by using the respective queue pair of the client; A method comprising:

2. The network device stores a correspondence between information about the plurality of clients and the access queue, and the step of transmitting the access requests of the plurality of clients to the access queue of the storage unit by the network device includes: The method of claim 1 , further comprising: transmitting, by the network device, the access requests of the plurality of clients to the access queue of the storage unit based on the correspondence relationship.

3. The information about the plurality of clients is connection information generated when the plurality of clients establish a connection to the network device, and the step of transmitting the access requests of the plurality of clients to an access queue of a storage unit by the network device includes: determining the access queue based on connection information corresponding to the client carried in the access request and the correspondence relationship when an access request of any one of the clients is received; sending the connection information and the access request to the access queue; Including, The processing result returned by the storage unit includes the connection information, and the step of returning the processing result of the access request returned by the storage unit to a client corresponding to the access request by the network device includes: determining, by the network device, the client corresponding to the access request based on the connection information, and returning the processing result to the client corresponding to the access request; 3. The method of claim 2, comprising:

4. The information about the plurality of clients is connection information generated when the plurality of clients establish a connection to the network device, and the step of transmitting the access requests of the plurality of clients to an access queue of a storage unit by the network device includes: when an access request for any one of the plurality of clients is received, assigning a local identifier to a client identifier carried in the access request, the local identifier uniquely identifying the client; and establishing a correspondence between the client identifier, the local identifier, and connection information corresponding to the client; replacing the client identifier carried in the access request with the local identifier; sending the access request to the access queue corresponding to the connection information; Including, The step of returning, by the network device, the processing result of the access request returned by the storage unit to a client corresponding to the access request, comprises:

3. The method of claim 2, further comprising: upon receiving the processing result of the access request returned by the storage unit, obtaining, by the network device, the local identifier from the processing result; determining the connection information corresponding to the client based on the local identifier; and returning the processing result to the client corresponding to the connection information.

5. The method according to claim 3 or 4, wherein a remote direct memory access (RDMA) connection is established between each of the plurality of clients and the network device, and a queue pair (QP) is generated when the RDMA connection is established.

6. A network device configured to communicate with each of a plurality of clients by using a respective queue pair, each queue pair being formed by a transmit queue of a client and a receive queue of the network device; a receiving unit configured to receive access requests sent by the plurality of clients connected to the network device by using the respective queue pairs; a sending unit configured to send the access request to an access queue of a storage unit; Including, The receiving unit is further configured to receive a processing result of the access request in the access queue, the processing result being returned by the storage unit after the storage unit executes the access request; The network device, wherein the sending unit is further configured to return the processing result of the access request returned by the storage unit to a client corresponding to the access request by using the respective queue pair of the client.

7. the network device further includes another storage unit; the other storage unit is configured to store a correspondence between information about the plurality of clients and the access queue; The transmitting unit: The network device of claim 6 , configured to send the access requests of the plurality of clients to the access queue of the storage unit based on the correspondence relationship.

8. The information about the plurality of clients is connection information generated when the plurality of clients respectively establish connections to the network device, the network device further comprising a processing unit; The processing unit is configured to, when receiving an access request from any one of the plurality of clients, determine the access queue based on connection information corresponding to the client carried in the access request and the correspondence relationship; the sending unit is configured to send the connection information and the access request to the access queue; The network device of claim 7 , wherein the sending unit is further configured to determine the client corresponding to the access request based on the connection information, and return a processing result to the client corresponding to the access request.

9. The information about the plurality of clients is connection information generated when the plurality of clients respectively establish connections to the network device, the network device further comprising a processing unit; The processing unit is configured to, when receiving an access request from any one of the plurality of clients, assign a local identifier to a client identifier carried in the access request, the local identifier uniquely identifying the client, and to establish a correspondence between the client identifier, the local identifier, and connection information corresponding to the client, and replace the client identifier carried in the access request with the local identifier; The sending unit is configured to send the access request to the access queue corresponding to the connection information; The processing unit is further configured, when receiving a processing result of the access request returned by the storage unit, to obtain the local identifier from the processing result, and to determine the connection information corresponding to the client based on the local identifier; The network device of claim 7 , wherein the sending unit is further configured to determine the client corresponding to the access request based on the connection information, and return the processing result to the client corresponding to the access request.

10. The network device of claim 8 or 9, wherein a remote direct memory access (RDMA) connection is established between each of the plurality of clients and the network device, and the connection information is a queue pair (QP) generated when the RDMA connection is established.

11. A computing device comprising a memory and a processor, the computing device being configured to perform the method of any one of claims 1 to 5 as a network device when the processor executes computer instructions stored in the memory.

12. 6. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor embedded in a network device, performs the functions of the method according to any one of claims 1 to 5.

13. A storage device including a network device and a storage unit, the network device is configured to communicate with each of a plurality of clients by using a respective queue pair, each queue pair being formed by a transmit queue of a client and a receive queue of the network device; the network device is further configured to receive access requests sent by the plurality of clients connected to the network device by using the respective queue pairs; the storage unit is connected to the plurality of clients using the network device, the network device being configured to transmit the access requests of the plurality of clients to an access queue of the storage unit; the storage unit is configured to execute the access requests in the access queue and return a result of processing the access requests; A storage device, wherein the network device is further configured to return the processing result of the access request returned by the storage unit to a client corresponding to the access request by using the respective queue pair of the client.

14. The storage device of claim 13, wherein the network device stores a correspondence between information about the plurality of clients and the access queue, and the network device is configured to transmit the access requests of the plurality of clients to the access queue of the storage unit based on the correspondence.

15. The information about the plurality of clients is connection information generated when the plurality of clients respectively establish connections to the network device, and when the network device is configured to transmit the access requests of the plurality of clients to the access queue of the storage unit, When receiving an access request from any one of the plurality of clients, determine the access queue based on connection information corresponding to the client carried in the access request and the correspondence relationship; and configured to transmit the connection information and the access request to the access queue; the storage unit returns the connection information when returning a processing result of the access request; The storage device according to claim 14 , wherein the network device determines the client corresponding to the access request based on the connection information, and returns the processing result to the client corresponding to the access request.

16. 16. The storage device of claim 15, wherein a remote direct memory access (RDMA) connection is established between each of the plurality of clients and the network device, and the connection information is a queue pair (QP) generated when the RDMA connection is established.

Images