Node access method and apparatus, and electronic apparatus

Abstract

Embodiments of the present disclosure provide a node access method and apparatus, and an electronic apparatus. The method comprises: extending a first access client by means of kernel extension technology, to obtain a second access client, wherein the second access client comprises a kernel-mode program, and the kernel-mode program is used for determining a plurality of first virtual sub-clients corresponding to the second access client; and accessing distributed storage nodes by means of the plurality of first virtual sub-clients. Use of the described technical solution solves the problem in the related art that existing NVMe clients cannot concurrently access a plurality of storage nodes in distributed storage.

Description

Node access method, device, and electronic device

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on June 28, 2023, with application number 2023107815017 and invention name “Node access method, device, and electronic device,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of communications, and in particular, to a node access method, device, and electronic device. Background Art

[0003] Currently, distributed storage systems provide a customized client interface by default. Client nodes access metadata servers and data servers simultaneously, enabling them to obtain the data distribution of the entire storage space. The client proxy all external requests and selects the target storage node through a related allocation algorithm to complete the actual read and write operations. The application layer does not need to understand the internal details of the storage cluster, achieving high-performance concurrent access to the storage cluster. However, the customized client interface of the storage cluster is difficult to efficiently access the existing operating system (OS) kernel storage process. Application-layer software needs to be adapted and modified accordingly to access storage, which greatly limits the promotion and application of distributed storage systems. Therefore, distributed storage clusters generally connect customized block storage to the Linux kernel through kernel modules such as Non-Volatile Memory Express over Fabrics (NVMe-oF) and Network Block Device (nbd).

[0004] The nbd module provides a unified method for Linux systems to access remote storage devices. Each nbd client establishes a link with a dedicated nbd server, mapping the remote storage device to the client's local device. This method has two drawbacks: First, the client can only access a specific nbd server, which then forwards the request, and the performance ceiling is heavily dependent on nbd's forwarding capabilities. Second, kernel modules are difficult to modify, requiring a kernel recompile and strictly bound to the kernel version. The RBD (RADOS Block Device) implemented by the Ceph Distributed File System (Ceph) has a similar implementation mechanism to nbd, but requires a new kernel module to implement the RADOS protocol. This strong binding to the OS kernel limits its application scenarios. The EBS (Elastic Block Storage) storage system uses the nbd module to implement remote access, but does not address the nbd module's shortcomings.

[0005] With the emergence of a new generation of storage media represented by solid-state drive (SSD) flash memory, distributed storage systems provide standard block storage devices through the NVMe-oF protocol, enabling massive data storage for databases, big data, cloud computing, and other businesses. However, by default, NVMe clients establish a link with an NVMe-oF Target service and access remote distributed storage through this link. As a result, a single NVMe-oF channel becomes the read and write bottleneck of the distributed storage cluster. The MultiPath function supported by the NVMe-oF protocol enables multiple network links to access the same Target service, avoiding the risk of single-node failure at the network layer. However, it cannot concurrently access multiple storage nodes of distributed storage, making it difficult to fully utilize the access performance of the storage system.

[0006] Regarding the problem in related technologies that existing NVMe clients cannot concurrently access multiple storage nodes of distributed storage, no effective solution has been proposed yet.

[0007] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology.

[0008] Summary of the Invention

[0009] The embodiments of the present disclosure provide a node access method, device, and electronic device to at least solve the problem that existing NVME clients cannot concurrently access multiple storage nodes of distributed storage.

[0010] According to one aspect of an embodiment of the present disclosure, a node access method is provided, comprising: extending a first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel-state program, and the kernel-state program is used to determine multiple first virtual sub-clients corresponding to the second access client; and accessing a distributed storage node through the multiple first virtual sub-clients.

[0011] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: when the second access client also includes a user-state program, determining the second virtual sub-client corresponding to the second access client based on the user-state program; and accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0012] In an exemplary embodiment, accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node includes: setting a timer for the second virtual sub-client; and obtaining metadata information in the metadata storage node within the timing duration according to the timing duration of the timer.

[0013] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method further includes: establishing links between the multiple first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

[0014] In an exemplary embodiment, accessing a distributed storage node through the multiple first virtual sub-clients includes: parsing a read / write request sent by a block device through the second access client to obtain a target address; determining a target node corresponding to the target address through an address mapping algorithm; and sending the read / write request to the target node, wherein the read / write request is sent through the first virtual sub-client linked to the target node.

[0015] In an exemplary embodiment, determining the target node corresponding to the target address through an address mapping algorithm includes: determining a hash value corresponding to the target address through the address mapping algorithm; and determining the target node corresponding to the hash value based on metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0016] In an exemplary embodiment, after sending the read / write request to the target node, the method further includes: receiving a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing the metadata storage node at the next moment after the current moment to obtain the latest metadata information in the metadata storage node.

[0017] In an exemplary embodiment, the multiple first virtual sub-clients include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; one first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0018] In an exemplary embodiment, sending the read / write request to the target node includes: if the virtual sub-client is a third virtual sub-client, transmitting the read / write request to the third virtual sub-client via the first virtual sub-client; converting the read / write request into a read / write request based on the second data transmission protocol via the third virtual sub-client, and sending the read / write request based on the second data transmission protocol to the target node. A response message to the request is then forwarded to the first virtual client, which completes the read / write operation.

[0019] According to another aspect of the embodiment of the present disclosure, a node access device is also provided, including: an extension module, configured to expand a first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine multiple first virtual sub-clients corresponding to the second access client; an access module, configured to access a distributed storage node through the multiple first virtual sub-clients.

[0020] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned node access method when running.

[0021] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the node access method through the computer program.

[0022] Through this disclosure, a first access client is expanded through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel-mode program that is used to determine multiple first virtual sub-clients corresponding to the second access client; and distributed storage nodes are accessed through the multiple first virtual sub-clients. In other words, by expanding the first access client through kernel extension technology, a second access client is obtained that can access distributed storage nodes through multiple first virtual sub-clients, solving the technical problem of existing NVMe clients being unable to concurrently access multiple storage nodes of distributed storage. This achieves the technical effect of concurrent access to distributed storage nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0024] FIG1 is a hardware structure block diagram of a computer terminal of a node access method according to an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of a node access method according to an embodiment of the present disclosure;

[0026] FIG3 is a schematic diagram of a control plane and a data plane of a node access method according to an embodiment of the present disclosure;

[0027] FIG4 is a schematic diagram (1) of a second access client of a node access method according to an embodiment of the present disclosure;

[0028] FIG5 is a schematic diagram of establishing a link according to a node access method according to an embodiment of the present disclosure;

[0029] FIG6 is a schematic diagram of a read and write process of a node access method according to an embodiment of the present disclosure;

[0030] FIG7 is a schematic diagram of metadata active update of a node access method according to an embodiment of the present disclosure;

[0031] FIG8 is a flowchart of a passive metadata update method of a node access method according to an embodiment of the present disclosure;

[0032] FIG9 is a schematic diagram (II) of a second access client of a node access method according to an embodiment of the present disclosure;

[0033] FIG10 is a structural block diagram of a node access device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The method embodiments provided in the embodiments of the present disclosure can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal of the node access method of the embodiment of the present disclosure. As shown in Figure 1, the computer terminal may include one or more (only one is shown in Figure 1) processors 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include more or fewer components than those shown in Figure 1, or have a different configuration with the same functions as those shown in Figure 1 or more functions than those shown in Figure 1.

[0037] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the node access method in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] FIG2 is a flow chart of a node access method according to an embodiment of the present disclosure, which is applied to the above-mentioned computer terminal. As shown in FIG2 , the method includes the following steps:

[0040] Step S202: Expand the first access client through kernel expansion technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client;

[0041] Optionally, the kernel extension technology used in the embodiments of the present disclosure includes, but is not limited to, the new kernel extension technology (extended Berkeley Packet Filter, eBPF). eBPF provides a mechanism for securely injecting extension code when kernel events and user program events occur, and is widely used in cloud native, performance tuning, hardware acceleration, and other fields.

[0042] Step S204: accessing the distributed storage node through the multiple first virtual sub-clients.

[0043] The disclosed embodiment, through the above steps, uses kernel extension technology to expand the first access client to obtain a second access client, wherein the second access client includes a kernel-mode program, and the kernel-mode program is used to determine multiple first virtual sub-clients corresponding to the second access client; and access the distributed storage node through the multiple first virtual sub-clients. In other words, by expanding the first access client through kernel extension technology, a second access client is obtained that can access the distributed storage node through multiple first virtual sub-clients, thereby resolving the technical problem that existing NVMe clients cannot concurrently access multiple storage nodes of distributed storage. This achieves the technical effect of concurrent access to distributed storage nodes.

[0044] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: when the second access client also includes a user-state program, determining the second virtual sub-client corresponding to the second access client based on the user-state program; and accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0045] It can be understood that the embodiment of the present disclosure injects the client function of distributed storage into the NVMe client process through kernel extension technology, so that the NVMe client can establish a link with all nodes of the storage cluster. Specifically, the expanded NVMe client maintains both the control plane and the data plane networks, as shown in Figure 3, wherein the data plane establishes a link with multiple distributed storage nodes through multiple first virtual sub-clients, as shown in Figure 4, the first virtual sub-client is equivalent to NVMeClient431, 432, 433 in Figure 4, the first virtual sub-client and the distributed storage node can be a one-to-one maintenance relationship, that is, one first virtual sub-client is linked to one distributed storage node; or a one-to-many maintenance relationship, that is, one first virtual sub-client is linked to multiple distributed storage nodes, but in the one-to-many case, the number of distributed storage nodes linked to the first virtual sub-client should be less than the number threshold, wherein the number threshold is the maximum number of distributed storage nodes that the first virtual sub-client can link to. If the number of distributed storage nodes linked to the first virtual sub-client exceeds the number threshold, the IO read and write request may not be forwarded in time. The data plane is maintained by a second virtual sub-client determined by a user-mode program. The second virtual sub-client is equivalent to the MetaClient in FIG4 . The second virtual sub-client can access the metadata storage node and obtain metadata information in the metadata node.

[0046] In an exemplary embodiment, accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node includes: setting a timer for the second virtual sub-client; and obtaining metadata information in the metadata storage node within the timing duration according to the timing duration of the timer.

[0047] Specifically, to allow the second access client to promptly obtain metadata changes in the distributed storage cluster, the second virtual sub-client can intermittently access the metadata storage node under certain restrictions. These restrictions include timing and / or quantity. Specifically, methods include: Method 1: Setting a timer to periodically obtain metadata information from the metadata storage node within a predetermined time period to update the metadata information in the second access client; Method 2: Setting an IO access request forwarding counter to access the metadata storage node to update the metadata information after the second access client has forwarded a pre-set number of IO read and write requests.

[0048] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method further includes: establishing links between the multiple first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

[0049] It can be understood that, as shown in Figure 5, step S501 first establishes a link between the second virtual sub-client and the metadata storage node; step S502 obtains metadata information; step S503 establishes a link between the first virtual sub-client and the distributed storage node. Specifically, the metadata information includes the current cluster information and data distribution information of the distributed storage cluster, wherein the current cluster information includes a storage server list, data partition information, address mapping relationship (or algorithm), etc. Based on the obtained metadata information, a link between the first virtual sub-client and the distributed storage node can be established, such as the NVMe-oF link shown in Figure 5, and the first virtual sub-client can forward read and write requests to the distributed storage node through the link. It should be noted that the link is preferably an NVMe-oF link.

[0050] In an exemplary embodiment, accessing a distributed storage node through the multiple first virtual sub-clients includes: parsing a read / write request sent by a block device through the second access client to obtain a target address; determining a target node corresponding to the target address through an address mapping algorithm; and sending the read / write request to the target node, wherein the read / write request is sent through the first virtual sub-client linked to the target node.

[0051] That is, the read / write request is sent to the target node through the first virtual sub-client linked to the target node, so that the target node performs the read / write operation indicated by the read / write request. It should be noted that the target address is used to indicate the address of the target node in multiple distributed storage nodes. Furthermore, for each of the multiple first virtual sub-clients, when it is determined that the target address is the address of the distributed storage node (i.e., the target node) linked to each of the first virtual sub-clients, each first virtual sub-client sends the read / write request to the target node, and the forwarding operations of the multiple first virtual sub-clients can be performed concurrently.

[0052] In an exemplary embodiment, determining the target node corresponding to the target address through an address mapping algorithm includes: determining a hash value corresponding to the target address through the address mapping algorithm; and determining the target node corresponding to the hash value based on metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0053] It is understandable that the block device sends the read / write request to the second access client, which parses the read / write request to obtain the target address of the target node to which the read / write request needs to be forwarded, wherein the target address includes the logical block address (LBA) and length information. The metadata information obtained by the second virtual sub-client is synchronized with the kernel state portion of the second access client through the maps mechanism. Furthermore, the second access client calculates the hash value corresponding to the target address based on the target address and the address mapping algorithm, and determines the target node corresponding to the read / write request from the metadata information based on the hash value. Preferably, the address mapping algorithm can adopt a hash algorithm. For example, as shown in Figure 6, the block device sends IO requests X and Y (equivalent to the IO read / write requests in the above embodiment), determines the target nodes DataServer0 and DataServer1 corresponding to IO requests X and Y respectively based on the maps mechanism and the address mapping algorithm, forwards the IO read / write request to the obtained target nodes, and ultimately obtains IO responses X and Y from DataServer0 and DataServer1 to the IO requests.

[0054] Optionally, sending the read and write request to the target node includes: determining the server corresponding to the target distributed storage node (i.e., the target node), wherein the server is only connected to the target distributed storage node, or the server is connected to multiple distributed storage nodes; receiving the read and write request sent by the first virtual sub-client through the server, and forwarding the read and write request to the target distributed storage node.

[0055] In an exemplary embodiment, after sending the read / write request to the target node, the method further includes: receiving a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing the metadata storage node at the next moment after the current moment to obtain the latest metadata information in the metadata storage node.

[0056] Furthermore, when the response address is different from the target address, the second access client instructs the second virtual sub-client to access the metadata storage node at the next moment after the current moment to obtain the latest metadata information in the metadata storage node, wherein the time difference between the current moment and the next moment is less than a preset threshold, and the second virtual sub-client is a second virtual sub-client determined by the user-mode program included in the second access client.

[0057] It is understandable that the second access client will periodically update the metadata information, that is, the above embodiment mentioned that the metadata information of the metadata storage node is obtained by the second virtual sub-client through a timer or other means at regular intervals. This is an active update solution, as shown in Figure 7, specifically, the metadata information in the metadata storage node (equivalent to the metadata server in Figure 7) is actively obtained by the second virtual sub-client (equivalent to the metadata client in Figure 7), and shared with the kernel-state NVMe client service through the maps mechanism. In addition, a passive update solution is also included, specifically: parsing the IO response result fed back by the received distributed storage node to obtain the response address. If the response address is the same as the target address indicated by the IO read / write request corresponding to the response result, the response node is determined to be the target node; if the response address is different from the target address indicated by the IO read / write request corresponding to the response result, the response node is determined to be the actual node, and the current metadata information needs to be updated, thereby triggering the passive update process. As shown in Figure 8, the kernel-state program notifies the second virtual sub-client that the metadata information needs to be updated immediately, and the second virtual sub-client will obtain the metadata information of the metadata storage node at the next moment of the current moment.

[0058] In an exemplary embodiment, the multiple first virtual sub-clients include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; one first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0059] As shown in FIG9 , the second access client may include another kernel-mode program 91, which includes a first virtual sub-client and two or more third virtual sub-clients (equivalent to Clients 911 and 912 in FIG9 ). The first virtual sub-client is based on a first data transmission protocol, preferably the NVMe protocol, and the third virtual sub-client is based on a second data transmission protocol, which is a customized data transmission protocol.

[0060] In an exemplary embodiment, sending the read / write request to the target node includes: in the case where the virtual sub-client is a third virtual sub-client, passing the read / write request to the third virtual sub-client through the first virtual sub-client; converting the read / write request into a read / write request based on the second data transmission protocol through the third virtual sub-client, and sending the read / write request based on the second data transmission protocol to the target node, wherein the third virtual client assists the first virtual client in achieving higher read / write efficiency.

[0061] Specifically, it can be understood that when the second access client uses kernel state 91 and needs to send an IO read / write request through the third virtual sub-client, the first virtual sub-client needs to pass the IO read / write request to the third virtual sub-client, which then converts the request into a read / write request based on the second data transmission protocol. When the second access client initially establishes a connection with the distributed storage node, the third virtual sub-client establishes an underlying network channel based on a custom data transmission protocol with the distributed storage node, and then sends the read / write request based on the second data transmission protocol to the target node through the underlying network channel.

[0062] Next, the node access method will be further described in conjunction with the following embodiments.

[0063] The emergence of the NVMe protocol has greatly improved the access performance of storage devices, and NVMe components have become the default kernel modules of Linux systems. This disclosure proposes a method for optimizing NVMe clients based on kernel extension technology (not limited to eBPF technology). It also proposes an innovative transformation solution for NVMe clients, enabling efficient multi-link concurrent access to distributed storage clusters, addressing the current performance bottleneck of NVMe clients accessing distributed storage systems remotely.

[0064] The new kernel extension technology represented by eBPF provides a method for non-kernel developers to control and modify the operation of kernel modules. The goal of the disclosed embodiment is to use the eBPF mechanism to inject customized logic into the NVMe kernel module, so that the NVMe client can timely obtain and update the distributed storage cluster metadata and storage space distribution information, and then calculate the target storage node to which the data belongs based on the target address of the IO request, and finally the target storage node directly completes the IO read and write operations. The solution provided by the disclosed embodiment has the following advantages: 1) It enables the NVMe client to directly access the target storage node, avoiding the forwarding overhead of a single proxy service and effectively reducing data read and write latency; 2) The NVMe client accesses multiple storage nodes at the same time, avoiding the single-link performance bottleneck and significantly improving the overall read and write performance of the storage system; 3) The kernel extension technology is used to inject customized code into the NVMe client module, without the need to modify the compiled kernel module, avoiding binding with the OS kernel, and without modifying the IO interface of the upper-layer application, which is suitable for various application scenarios.

[0065] Specifically, the disclosed embodiment injects distributed storage client functionality into the NVMe client process through kernel extension technology, enabling the NVMe client to establish connections with all nodes in the storage cluster. As shown in Figure 3, the client maintains both the control plane and data plane networks. The control plane network connects to the metadata server MetaServer to obtain cluster metadata information, with low requirements for network bandwidth and latency. The data plane network establishes links with multiple storage servers DataServer, carrying the NVMe-oF protocol and enabling remote data reading and writing.

[0066] Optionally, when the storage cluster is too large, it supports multiple data servers sharing the same data link, and the storage server connected by the client completes a small range of proxy forwarding to avoid the client maintaining a large number of concurrent links.

[0067] Furthermore, the schematic diagram of the NVMe client and server composition in the embodiment of the present disclosure is shown in Figure 4, which specifically includes a block device 41; a user state 42, the user state 42 includes a metadata client MetaClient421 (equivalent to the second virtual sub-client in the above embodiment), and MetaClient421 is linked to the metadata server MetaServer; a kernel state NVMeCltService43, and the kernel state 43 includes NVMe clients NVMeClient431, 432, and 433 (equivalent to the first virtual sub-client in the above embodiment), and NVMeClient431, 432, and 433 are respectively linked to DataServer46, 47, and 48 through NVMeTarget.

[0068] MetaClient establishes a link with the storage cluster to obtain the latest cluster status and data distribution information in a timely manner, and synchronizes it to the kernel-mode NVMeCltService module using a shared maps mechanism. Taking the distributed storage project Ceph as an example, given the cluster's PG list and OSD grouping information, the OSD server where the target address is located can be calculated based on the access target address LBA using a fixed hash algorithm. Therefore, by writing the Ceph cluster's PG and OSD list information into the maps table and implementing the hash calculation interface on the client service, the routing calculation capabilities of the Ceph cluster can be realized.

[0069] NVMeCltService also includes user-injected code NVMeFunc434. NVMeFunc modifies the read and write processes of the kernel NVMe module and manages multiple NVMeClient modules (i.e., NVMe clients 431-433 in Figure 4, but Figure 4 is only an example, i.e., NVMe clients are not limited to the three in Figure 4; there can be more) to complete IO operations. Each data storage server runs an NVMe-oF Target service and registers the same Subsystem, Namespace, and other resources. NVMeClient and NVMe-oF Target are linked one by one. All clients access the same global logical storage volume, return the same global logical storage volume information to the kernel, and map it to an NVMe logical block device (equivalent to block device 41 in Figure 4).

[0070] The read and write operations of the local block device (equivalent to the block device 41 in Figure 4) trigger NVMe requests. NVMeCltService parses the target address of the request, calculates the server where the target address is located based on the data distribution information, and selects the corresponding NVMeClient to send read and write requests to the target data server. After the IO operation is completed, NVMeClient receives the IO response and returns it to the kernel's NVMe driver interface through the same NVMe session.

[0071] Furthermore, the specific chain building process is shown in Figure 5, which includes the following steps:

[0072] Step S501: The client establishes a link with the MetaServer through the connect command;

[0073] Step S502: Obtain metadata information;

[0074] Step S503: Establish an NVMe-oF link with DataServe.

[0075] Specifically, the standard nvme client establishes a link with the server through the following command:

[0076] nvme connect-t nettype-n"nqn.XX.yy.mm"-a IPAddr-s port;

[0077] The nvme client of the embodiment of the present disclosure supports establishing links with multiple nodes at the same time. The command format is as follows:

[0078] nvme connect-t nettype-n"nqn.XX.yy.mm"-m MetaSvr:port;

[0079] "-m MetaSvr:port" specifies the MetaServer address and port information, as shown in Figure 5. The client uses the connect command to establish a connection with the MetaServer and obtain current cluster information, including a list of storage servers, data partition information, and address mapping relationships (or algorithms). Based on the obtained storage server information, the client establishes NVMe-oF links to multiple storage service nodes. Each storage server needs to start the NVMe-oF Target service, register the same Target resource, and support request routing and forwarding. If the storage server finds that the request address does not belong to this service, it can forward the request to the actual destination server.

[0080] Furthermore, the business application's access to the logical block device triggers the NVMe client's read and write process, as shown in Figure 6, including:

[0081] 1) The NVMeClient client first obtains the target address from the read and write request (such as IO request X and IO request Y in Figure 6), and obtains the logical address LBA and length Len information by parsing the NVMe request message;

[0082] 2) Based on the cluster composition information cached in the maps, such as the cluster partition table and data server list, a hash algorithm is used to calculate the storage server DataSever where the LBA address is located. The NVMe client object that establishes a link with this server is selected, and an NVMe-oF protocol-type request message is assembled and sent to the target node. This message carries information such as the read and write LBA, Len, operation type, and session number. For example, in Figure 6, based on the maps mechanism, NVMe request X is sent to DataSever0, and NVMe request Y is sent to DataSever1.

[0083] 3) After the server completes the reading and writing, it constructs a response message of the NVMe-oF protocol type and returns it to the NVMeClient client, such as NVMe response X and NVMe response Y in Figure 5;

[0084] 4) The NVMeClient client reuses the same NVMe link ID to return the IO response result to the application layer, such as IO response X and IO response Y in Figure 6. The NVMe link ID belongs to the NVMe session ID created and used by the operating system layer, which is used to identify the NVMe link at the operating system level.

[0085] Furthermore, the metadata update process is as follows, including two update modes: active update and passive update. Specifically:

[0086] The client's MetaClient is responsible for reading the latest cluster metadata information from the cluster metadata server and saving it to the DataMap table. The user-state MetaClient module and the kernel-state NVMeCltService implement data interaction through the shared DataMap. MetaClient supports two methods: active update and passive update. Active update means that the client obtains cluster metadata at regular intervals, as shown in Figure 7. It includes the following steps:

[0087] Step S701: The second virtual sub-client obtains metadata information from the metadata storage node at a regular interval;

[0088] Step S702: The second virtual sub-client obtains metadata information;

[0089] Step S703: The user state where the second virtual sub-client is located shares metadata information with the kernel state through a maps mechanism.

[0090] Passive update means that when the client NVMeCltService finds that the source address of the response message (equivalent to the response result in the above embodiment) does not match the target server of the request message, it triggers the metadata passive update process and notifies MetaClient to re-acquire the cluster metadata. Among them, DataSever supports forwarding requests whose target address is not its own to the correct server. The specific process is shown in Figure 8, including:

[0091] Step S801: In kernel state, the source address of the response result does not match the target node of the IO read / write request, triggering a passive update of metadata;

[0092] Step S802: The second virtual sub-client obtains metadata information from the metadata storage node at a time next to the current time;

[0093] Step S803: The second virtual sub-client obtains metadata information;

[0094] Step S804: The user state where the second virtual sub-client is located shares metadata information with the kernel state through a maps mechanism.

[0095] Optionally, there is another alternative for the data plane link in the embodiment of the present disclosure. As shown in Figure 9, the client and the storage cluster only create one complete NVMe-oF link, and other links use a custom data transmission protocol. The client in the alternative also includes clients 911 and 912 (equivalent to the third virtual sub-client in the above embodiment), which transmit data through the underlying network channel (that is, not through the NVMe-oF Target). The client converts the NVMe request into a custom internal IO request and sends it to the target storage server. The internal IO response is encapsulated as a standard NVMe response and returned to the NVMeCltService service, that is, the client connects to an NVMe-oF Target server through an NVMe-oF link, but supports multiple underlying network channels to transmit data. The alternative helps to achieve higher IO performance.

[0096] In summary, the NVMe client command interface of the embodiment of the present disclosure needs to specify the IP and port of the metadata server, which has obvious characteristics. In addition, the client of the embodiment of the present disclosure establishes a link channel with the metadata server and storage server in the storage cluster, and the network characteristics are very significant.

[0097] Furthermore, the disclosed embodiments extend the NVMe client to support multi-link concurrent access to the storage cluster, resolving the performance bottleneck of a single client in a distributed storage cluster and addressing the inability of customized clients to efficiently access kernel IO drivers. By adapting the kernel extension mechanism to the NVMe client, the solution eliminates the need to directly modify kernel modules, making implementation and deployment more flexible and resolving the kernel-binding issue of current implementations.

[0098] In this embodiment, a node access device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0099] FIG10 is a block diagram of a node access device according to an embodiment of the present disclosure. As shown in FIG10 , the node access device includes:

[0100] An extension module 1002 is configured to extend the first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client;

[0101] The access module 1004 is configured to access the distributed storage node through the multiple first virtual sub-clients.

[0102] The disclosed embodiment, through the above steps, uses kernel extension technology to expand the first access client to obtain a second access client, wherein the second access client includes a kernel-mode program, and the kernel-mode program is used to determine multiple first virtual sub-clients corresponding to the second access client; and access the distributed storage node through the multiple first virtual sub-clients. In other words, by expanding the first access client through kernel extension technology, a second access client is obtained that can access the distributed storage node through multiple first virtual sub-clients, thereby resolving the technical problem that existing NVMe clients cannot concurrently access multiple storage nodes of distributed storage. This achieves the technical effect of concurrent access to distributed storage nodes.

[0103] In an exemplary embodiment, the access module 1004 is further configured to, when the second access client also includes a user-state program, determine the second virtual sub-client corresponding to the second access client according to the user-state program; and access the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0104] In an exemplary embodiment, the access module 1004 is further configured to set a timer for the second virtual sub-client; and obtain metadata information in the metadata storage node within the timer duration according to the timer duration.

[0105] In an exemplary embodiment, the access module 1004 is further configured to respectively establish links between the multiple first virtual sub-clients and the distributed storage nodes, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

[0106] In an exemplary embodiment, the access module 1004 is further configured to parse the read and write requests sent by the block device through the second access client to obtain a target address; determine the target node corresponding to the target address through an address mapping algorithm; and send the read and write requests to the target node, wherein the read and write requests are sent through the first virtual sub-client linked to the target node.

[0107] In an exemplary embodiment, the access module 1004 is further configured to determine the hash value corresponding to the target address through the address mapping algorithm; and determine the target node corresponding to the hash value based on metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0108] In an exemplary embodiment, the access module 1004 is also configured to receive a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing the metadata storage node at the next moment of the current moment to obtain the latest metadata information in the metadata storage node.

[0109] In an exemplary embodiment, the extension module 1002 is further configured so that the multiple first virtual sub-clients include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; one first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0110] In an exemplary embodiment, the access module 1004 is further configured to, when the virtual sub-client is a third virtual sub-client, pass the read and write request to the third virtual sub-client through the first virtual sub-client; convert the read and write request into a read and write request based on the second data transmission protocol through the third virtual sub-client, and send the read and write request based on the second data transmission protocol to the target node.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present disclosure.

[0112] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0113] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0114] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0115] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0116] S1, expanding a first access client through kernel expansion technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine multiple first virtual sub-clients corresponding to the second access client;

[0117] S2. Access the distributed storage node through the multiple first virtual sub-clients.

[0118] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0119] Optionally, in this embodiment, the above-mentioned electronic device can also be configured to execute the above-mentioned step S1 through a computer program, expand the first access client through kernel extension technology, and obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine multiple first virtual sub-clients corresponding to the second access client; S2, access the distributed storage node through the multiple first virtual sub-clients.

[0120] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0121] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0122] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A node access method, comprising: Expanding the first access client by kernel expansion technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client; The distributed storage node is accessed through the multiple first virtual sub-clients.

2. The node access method according to claim 1, wherein, before accessing the distributed storage node through the plurality of first virtual sub-clients, the method further comprises: In the case where the second access client further includes a user state program, determining a second virtual sub-client corresponding to the second access client according to the user state program; The metadata storage node is accessed through the second virtual sub-client to obtain metadata information in the metadata storage node.

3. The node access method according to claim 2, wherein accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node comprises: Setting a timer for the second virtual sub-client; The metadata information in the metadata storage node within the timing duration is acquired according to the timing duration of the timer.

4. The node access method according to claim 1, wherein, before accessing the distributed storage node through the plurality of first virtual sub-clients, the method further comprises: Establishing links between the plurality of first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

5. The node access method according to claim 1, wherein: Accessing the distributed storage nodes through the multiple first virtual sub-clients includes: Parsing the read and write request sent by the block device through the second access client to obtain a target address; Determine the target node corresponding to the target address by an address mapping algorithm; The read / write request is sent to the target node, wherein the read / write request is sent through a first virtual sub-client linked to the target node.

6. The node access method according to claim 5, wherein after sending the read / write request to the target node, the method further comprises: Receive a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node and an actual node, wherein the actual node is a node to which the read / write request actually belongs when the read / write request does not belong to the target node; Parsing the response result to obtain a response address of the response result; In the case that the response address is different from the target address, the metadata storage node is accessed at a time next to the current time to obtain the latest metadata information in the metadata storage node.

7. The node access method according to claim 5, comprising: The plurality of first virtual sub-clients include at least one of the following: a plurality of first virtual sub-clients determined based on a first data transmission protocol; A first virtual sub-client determined based on the first data transmission protocol and a plurality of third virtual sub-clients determined based on the second data transmission protocol.

8. The node access method according to claim 7, wherein sending the read / write request to the target node comprises: In the case where the virtual sub-client is a third virtual sub-client, the read / write request is transmitted to the third virtual sub-client through the first virtual sub-client; The read / write request is converted into a read / write request based on the second data transmission protocol through the third virtual sub-client, and the read / write request based on the second data transmission protocol is sent to the target node.

9. A node access device, comprising: An extension module, configured to extend the first access client through a kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client; The access module is configured to access the distributed storage node through the multiple first virtual sub-clients.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images