Node access method and device and electronic device

Abstract

The embodiment of the invention provides a node access method and device and an electronic device.The method comprises the steps that a first access client side is expanded through a kernel expansion technology, a second access client side is obtained, the second access client side comprises a kernel mode program, and the kernel mode program comprises a kernel mode program; the kernel mode program is used for determining a plurality of first virtual sub-clients corresponding to the second access client; and accessing the distributed storage node through the plurality of first virtual sub-clients. By adopting the technical scheme, the problem that an existing NVME client cannot concurrently access a plurality of storage nodes of distributed storage in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular, to a node access method, device, and electronic device. Background Art

[0002] At present, the distributed storage system provides a customized client interface by default. The client node accesses the metadata server and the data server at the same time, and can obtain the data distribution of the entire storage space. The client agent all external requests and selects the target storage node through the relevant allocation algorithm to complete the real read and write operations. The application layer does not need to understand the internal details of the storage cluster, and realizes the high-performance concurrent access capability of the storage cluster. However, the customized client interface of the storage cluster is difficult to efficiently access the existing operating system (Operating System, referred to as OS) kernel storage process, and the application layer software needs to be adapted and modified accordingly to access the storage, which greatly limits the promotion and application of distributed storage systems. Therefore, distributed storage clusters generally connect customized block storage to the Linux kernel through kernel modules such as non-volatile memory expressover fabrics (NVMe-oF) and network block devices (nbd).

[0003] The nbd module provides a unified method for Linux systems to access remote storage devices. Each nbd client establishes a link with a dedicated nbd server and maps the remote storage device to the client. This method has two disadvantages: first, the client can only access a certain nbd server, and then the nbd server completes the request forwarding. The performance limit is heavily dependent on the forwarding capability of nbd; second, the kernel module is difficult to modify, and the kernel needs to be recompiled, and it is strictly bound to the kernel version. The RBD (RADOS BlockDevice) device implemented by the distributed storage system (Ceph Distributed File System, referred to as Ceph) has a similar implementation mechanism to nbd, and the client needs to add a kernel module to implement the RADOS protocol. The strong binding with the OS kernel limits the application scenario. The EBS (Elastic Block Storage) storage system implements remote access based on the nbd module, and does not solve the defects of the nbd module.

[0004] With the emergence of a new generation of storage media represented by solid state drive (SSD) flash memory, distributed storage systems provide standard block storage devices through the NVMe-oF protocol to achieve massive data storage for databases, big data, cloud computing and other businesses. However, the NVMe client establishes a link with an NVMe-oF Target service by default and accesses remote distributed storage through this link. Therefore, a single NVMe-oF channel becomes the read and write bottleneck of the distributed storage cluster. The MultiPath function supported by the NVMe-oF protocol enables multiple network links to access the same Target service, avoiding the risk of single-node failure at the network layer, but it cannot concurrently access multiple storage nodes of distributed storage, making it difficult to fully utilize the access performance of the storage system.

[0005] Regarding the problem in the related technology that the existing NVMe client cannot concurrently access multiple storage nodes of distributed storage, no effective solution has been proposed yet.

[0006] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0007] The embodiments of the present application provide a node access method, device, and electronic device to at least solve the problem that the existing NVME client cannot concurrently access multiple storage nodes of distributed storage.

[0008] According to one aspect of an embodiment of the present application, a node access method is provided, comprising: extending a first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client; and accessing a distributed storage node through the plurality of first virtual sub-clients.

[0009] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: when the second access client also includes a user state program, determining the second virtual sub-client corresponding to the second access client according to the user state program; accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0010] In an exemplary embodiment, accessing a metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node includes: setting a timer for the second virtual sub-client; and obtaining metadata information in the metadata storage node within the timing duration according to the timing duration of the timer.

[0011] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: establishing links between the multiple first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, and wherein the metadata information is obtained by the second access client from the metadata storage node.

[0012] In an exemplary embodiment, accessing a distributed storage node through the multiple first virtual sub-clients includes: parsing a read / write request sent by a block device through the second access client to obtain a target address; determining a target node corresponding to the target address through an address mapping algorithm; and sending the read / write request to the target node, wherein the read / write request is sent through the first virtual sub-client linked to the target node.

[0013] In an exemplary embodiment, determining the target node corresponding to the target address through an address mapping algorithm includes: determining a hash value corresponding to the target address through the address mapping algorithm; determining the target node corresponding to the hash value based on metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0014] In an exemplary embodiment, after sending the read / write request to the target node, the method further includes: receiving a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing a metadata storage node at the next moment of the current moment to obtain the latest metadata information in the metadata storage node.

[0015] In an exemplary embodiment, the multiple first virtual sub-clients include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; one first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0016] In an exemplary embodiment, sending the read / write request to the target node includes: in a case where the virtual sub-client is a third virtual sub-client, passing the read / write request to the third virtual sub-client through the first virtual sub-client; converting the read / write request into a read / write request based on the second data transmission protocol through the third virtual sub-client, and sending the read / write request based on the second data transmission protocol to the target node.

[0017] According to another aspect of the embodiment of the present application, a node access device is also provided, including: an extension module, used to expand a first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine multiple first virtual sub-clients corresponding to the second access client; an access module, used to access a distributed storage node through the multiple first virtual sub-clients.

[0018] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned node access method when running.

[0019] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the node access method through the computer program.

[0020] Through this application, the first access client is extended by kernel extension technology to obtain the second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine the multiple first virtual sub-clients corresponding to the second access client; and the distributed storage node is accessed through the multiple first virtual sub-clients. In other words, the first access client is extended by kernel extension technology to obtain the second access client that can access the distributed storage node through multiple first virtual sub-clients, which solves the technical problem that the existing NVMe client cannot concurrently access multiple storage nodes of distributed storage. The technical effect of concurrent access to distributed storage nodes is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The exemplary embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 It is a hardware structure block diagram of a computer terminal of the node access method of an embodiment of the present application;

[0023] Figure 2 is a flowchart of a node access method according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of the control plane and data plane of the node access method according to an embodiment of the present application;

[0025] Figure 4 Schematic diagram (I) of a second access client according to a node access method according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of establishing a link according to a node access method according to an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a read and write process of a node access method according to an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of active metadata update of a node access method according to an embodiment of the present application;

[0029] Figure 8 is a metadata passive update flow chart of a node access method according to an embodiment of the present application;

[0030] Fig. 9 Schematic diagram (II) of a second access client according to a node access method according to an embodiment of the present application;

[0031] Fig.10 It is a structural block diagram of a node access device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal of the node access method of an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the node access method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0036] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] Figure 2 is a flowchart of a node access method according to an embodiment of the present application, which is applied to the above-mentioned computer terminal, such as Figure 2 As shown, the steps of the method include:

[0038] Step S202, expanding the first access client through kernel expansion technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client;

[0039] Optionally, the kernel extension technology used in the embodiments of the present application includes but is not limited to a new kernel extension technology (extended Berkeley Packet Filter, referred to as eBPF). eBPF provides a mechanism for safely injecting extension code when kernel events and user program events occur, and is widely used in cloud native, performance tuning, hardware acceleration and other fields.

[0040] Step S204: accessing the distributed storage node through the multiple first virtual sub-clients.

[0041] The embodiment of the present application, through the above steps, expands the first access client through the kernel extension technology to obtain the second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine the multiple first virtual sub-clients corresponding to the second access client; and access the distributed storage node through the multiple first virtual sub-clients. In other words, by expanding the first access client through the kernel extension technology, a second access client that can access the distributed storage node through multiple first virtual sub-clients is obtained, which solves the technical problem that the existing NVMe client cannot concurrently access multiple storage nodes of the distributed storage. The technical effect of concurrent access to the distributed storage nodes is achieved.

[0042] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: when the second access client also includes a user state program, determining the second virtual sub-client corresponding to the second access client according to the user state program; accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0043] It is understandable that the embodiment of the present application injects the client function of distributed storage into the NVMe client process through kernel extension technology, so that the NVMe client can establish a link with all nodes of the storage cluster. Specifically, the extended NVMe client maintains both the control plane and the data plane networks, such as Figure 3 As shown, the data plane establishes links with multiple distributed storage nodes through multiple first virtual sub-clients, such as Figure 4 As shown, the first virtual sub-client is equivalent to Figure 4 In NVMeClient431, 432, and 433, the first virtual sub-client and the distributed storage node can be in a one-to-one maintenance relationship, that is, one first virtual sub-client is linked to one distributed storage node; or a one-to-many maintenance relationship, that is, one first virtual sub-client is linked to multiple distributed storage nodes. However, in the one-to-many case, the number of distributed storage nodes linked to the first virtual sub-client should be less than the number threshold, wherein the number threshold is the maximum number of distributed storage nodes that the first virtual sub-client can link to. If the number of distributed storage nodes linked to the first virtual sub-client exceeds the number threshold, the IO read and write request may not be forwarded in time. Among them, the data plane is maintained by the second virtual sub-client determined by the user-mode program, and the second virtual sub-client is equivalent to Figure 4 The second virtual sub-client can access the metadata storage node and obtain the metadata information in the metadata node.

[0044] In an exemplary embodiment, accessing a metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node includes: setting a timer for the second virtual sub-client; and obtaining metadata information in the metadata storage node within the timing duration according to the timing duration of the timer.

[0045] That is, in order to allow the second access client to obtain the change of the metadata information of the distributed storage cluster, the metadata storage node can be intermittently accessed through the second virtual sub-client under certain restrictions. The certain restrictions include: timing and / or quantity, and the specific methods include: Method 1, by setting a timer to regularly obtain the metadata information of the metadata storage node within the timing period, so as to update the metadata information in the second access client; Method 2: by setting an IO access request forwarding counter, after the second access client forwards a pre-designed number of IO read and write requests, the metadata storage node is accessed to update the metadata information.

[0046] In an exemplary embodiment, before accessing the distributed storage node through the multiple first virtual sub-clients, the method also includes: establishing links between the multiple first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, and wherein the metadata information is obtained by the second access client from the metadata storage node.

[0047] It is understandable that if Figure 5 As shown, step S501, first establish a link between the second virtual sub-client and the metadata storage node; step S502, obtain metadata information; step S503, establish a link between the first virtual sub-client and the distributed storage node. Specifically, the metadata information includes the current cluster information and data distribution information of the distributed storage cluster, wherein the current cluster information includes a storage server list, data partition information, address mapping relationship (or algorithm), etc. Based on the obtained metadata information, a link between the first virtual sub-client and the distributed storage node can be established, for example Figure 5 The NVMe-oF link shown, the first virtual sub-client can forward the read and write request to the distributed storage node through the link. It should be noted that the link is preferably an NVMe-oF link.

[0048] In an exemplary embodiment, accessing a distributed storage node through the multiple first virtual sub-clients includes: parsing a read / write request sent by a block device through the second access client to obtain a target address; determining a target node corresponding to the target address through an address mapping algorithm; and sending the read / write request to the target node, wherein the read / write request is sent through the first virtual sub-client linked to the target node.

[0049] That is, the read / write request is sent to the target node through the first virtual sub-client linked to the target node, so that the target node performs the read / write operation indicated by the read / write request. It should be noted that the target address is used to indicate the address of the target node in multiple distributed storage nodes. Further, for each of the multiple first virtual sub-clients, when it is determined that the target address is the address of the distributed storage node (i.e., the target node) linked to each of the first virtual sub-clients, each first virtual sub-client sends the read / write request to the target node, and the forwarding operations of the multiple first virtual sub-clients can be performed concurrently.

[0050] In an exemplary embodiment, determining the target node corresponding to the target address through an address mapping algorithm includes: determining a hash value corresponding to the target address through the address mapping algorithm; determining the target node corresponding to the hash value based on metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0051] It is understandable that the block device sends the read / write request to the second access client, and the second access client parses the target address of the target node to which the read / write request needs to be forwarded from the read / write request, wherein the target address includes the logical address (Logical Block Address, abbreviated as LBA) and length information. The metadata information obtained by the second virtual sub-client will synchronize the kernel state part of the second access client through the maps mechanism. Furthermore: the second access client calculates the hash value corresponding to the target address according to the target address and the address mapping algorithm, and determines the target node corresponding to the read / write request from the metadata information based on the hash value. Preferably, the address mapping algorithm can adopt a Hash algorithm. For example, Figure 6 As shown in , the block device sends IO requests X and Y (equivalent to the IO read and write requests in the above embodiment), determines the target nodes DataServer0 and DataServer1 corresponding to the IO requests X and Y respectively based on the maps mechanism and address mapping algorithm, forwards the IO read and write requests to the obtained target nodes and finally obtains the IO responses X and Y of DataServer0 and DataServer1 to the IO requests.

[0052] Optionally, sending the read and write request to the target node includes: determining a server corresponding to the target distributed storage node (i.e., the target node), wherein the server is only connected to the target distributed storage node, or the server is connected to multiple distributed storage nodes; receiving the read and write request sent by the first virtual sub-client through the server, and forwarding the read and write request to the target distributed storage node.

[0053] In an exemplary embodiment, after sending the read / write request to the target node, the method further includes: receiving a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing a metadata storage node at the next moment of the current moment to obtain the latest metadata information in the metadata storage node.

[0054] Furthermore, when the response address is different from the target address, the second access client instructs the second virtual sub-client to access the metadata storage node at the next moment after the current moment to obtain the latest metadata information in the metadata storage node, wherein the time difference between the current moment and the next moment is less than a preset threshold, and the second virtual sub-client is a second virtual sub-client determined by a user-mode program included in the second access client.

[0055] It is understandable that the second access client will periodically update the metadata information, that is, the above embodiment mentioned that the metadata information of the metadata storage node is obtained by the second virtual sub-client periodically through a timer or the like. This is an active update solution, such as Figure 7 As shown, specifically, through the second virtual sub-client (equivalent to Figure 7 The metadata client in the metadata storage node (equivalent to Figure 7 In addition, a passive update scheme is also included, which is as follows: the IO response result fed back by the distributed storage node is parsed to obtain the response address. If the response address is the same as the target address indicated by the IO read / write request corresponding to the response result, the response node is determined to be the target node; if the response address is different from the target address indicated by the IO read / write request corresponding to the response result, the response node is determined to be the actual node, and the current metadata information needs to be updated, thereby triggering the passive update process, such as Figure 8 As shown, the kernel state program informs the second virtual sub-client that the metadata information needs to be updated immediately, and the second virtual sub-client will obtain the metadata information of the metadata storage node at the next moment of the current moment.

[0056] In an exemplary embodiment, the multiple first virtual sub-clients include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; one first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0057] like Fig. 9 As shown, the second access client may include another kernel state program 91, which includes a first virtual sub-client and two or more third virtual sub-clients (equivalent to Fig. 9 The first virtual sub-client is based on a first data transmission protocol, preferably an NVMe protocol, and the third virtual sub-client is based on a second data transmission protocol, which is a custom data transmission protocol.

[0058] In an exemplary embodiment, sending the read / write request to the target node includes: in a case where the virtual sub-client is a third virtual sub-client, passing the read / write request to the third virtual sub-client through the first virtual sub-client; converting the read / write request into a read / write request based on the second data transmission protocol through the third virtual sub-client, and sending the read / write request based on the second data transmission protocol to the target node.

[0059] That is, it can be understood that, when the second access client adopts kernel state 91, when it is necessary to send IO read and write requests through the third virtual sub-client, the first virtual sub-client needs to pass the IO read and write requests to the third virtual sub-client, and the third virtual sub-client converts them into read and write requests based on the second data transmission protocol. When the second access client initially establishes a link with the distributed storage node, the third virtual sub-client and the distributed storage node establish an underlying network channel based on a custom data transmission protocol, and then the read and write requests based on the second data transmission protocol are sent to the target node through the underlying network channel.

[0060] Next, the node access method is further described in conjunction with the following embodiments.

[0061] The emergence of the NVMe protocol has greatly improved the access performance of storage devices, and the NVMe component has become the default kernel module of the Linux system. The embodiment of the present application proposes a method for optimizing the NVMe client based on kernel extension technology (not limited to eBPF technology), proposes an innovative transformation plan for the NVMe client, realizes multi-link concurrent and efficient access to the distributed storage cluster, and solves the performance bottleneck of the current NVMe client remote access to the distributed storage system.

[0062] The new kernel extension technology represented by eBPF provides a method for non-kernel developers to control and modify the operation of kernel modules. The goal of the embodiment of the present application is to use the eBPF mechanism to inject customized logic into the NVMe kernel module, so that the NVMe client can timely obtain and update the distributed storage cluster metadata and storage space distribution information, and then calculate the target storage node to which the data belongs according to the target address of the IO request, and finally the target storage node directly completes the IO read and write operation. The solution provided by the embodiment of the present application has the following advantages: 1) The NVMe client directly accesses the target storage node, avoiding the forwarding overhead of a single proxy service and effectively reducing data read and write delays; 2) The NVMe client accesses multiple storage nodes at the same time, avoiding the single-link performance bottleneck, and significantly improving the overall read and write performance of the storage system; 3) The kernel extension technology is used to inject custom code into the NVMe client module, without modifying the compiled kernel module, avoiding binding with the OS kernel, and without modifying the IO interface of the upper application, which is suitable for various application scenarios.

[0063] Specifically, the embodiment of the present application injects the client function of distributed storage into the NVMe client process through kernel extension technology, so that the NVMe client can establish a link with all nodes of the storage cluster. Figure 3 As shown in the figure, the client maintains two networks, the control plane and the data plane. The control plane network is linked to the metadata server MetaServer to obtain cluster metadata information, and has low requirements for network bandwidth and latency. The data plane network is linked to multiple storage servers DataServer, carrying the NVMe-oF protocol to complete remote data reading and writing and other functions.

[0064] Optionally, when the storage cluster is too large, it supports multiple data servers to share the same data link, and the storage server connected by the client completes a small range of proxy forwarding to avoid the client maintaining a large number of concurrent links.

[0065] Further, the schematic diagram of the NVMe client and server in the embodiment of the present application is as follows Figure 4 As shown, it specifically includes a block device 41; a user state 42, the user state 42 includes a metadata client MetaClient421 (equivalent to the second virtual sub-client in the above embodiment), and MetaClient421 is linked to a metadata server MetaServer; a kernel state NVMeCltService43, the kernel state 43 includes NVMe clients NVMeClient431, 432, 433 (equivalent to the first virtual sub-client in the above embodiment), and NVMeClient431, 432, 433 are linked to DataServer46, 47, 48 respectively through NVMeTarget.

[0066] MetaClient establishes a link with the storage cluster to obtain the latest cluster status and data distribution information in a timely manner, and synchronizes it to the kernel-state NVMeCltService module using the shared maps mechanism. Taking the distributed storage project Ceph as an example, the PG list of the cluster and the grouping information of the OSD are known. The OSD server where the target address is located can be calculated based on the access target address LBA through a fixed Hash algorithm. Therefore, the PG and OSD list information of the Ceph cluster is written into the maps table. At the same time, the client service implements the Hash calculation interface, which can realize the routing calculation capability of the Ceph cluster.

[0067] NVMeCltService also includes user injection code NVMeFunc434, NVMeFunc modifies the read and write process of the kernel NVMe module, manages multiple NVMeClient modules (i.e. Figure 4 NVMe clients 431-433 in, but Figure 4 Just as an example, NVMe clients are not limited to Figure 4 3 of them, and there can be more) to complete IO operations. Each data storage server will run an NVMe-oF Target service and register the same Subsystem, Namespace and other resources. NVMeClient and NVMe-oF Target are linked one by one. All clients access the same global logical storage volume, return the same global logical storage volume information to the kernel, and map it to an NVMe logical block device (equivalent to Figure 4 Block device 41 in.

[0068] Local block device (equivalent to Figure 4 The read and write operations of the block device 41 in trigger the NVMe request. NVMeCltService parses the target address of the request, calculates the server where the target address is located according to the data distribution information, and selects the corresponding NVMeClient to send the read and write request to the target data server. After the IO operation is completed, NVMeClient receives the IO response and returns it to the kernel's NVMe driver interface through the same NVMe session.

[0069] Further, the specific chain building process is as follows Figure 5 As shown, the following steps are included:

[0070] Step S501: The client establishes a link with the MetaServer through the connect command;

[0071] Step S502: Obtain metadata information;

[0072] Step S503: Establish an NVMe-oF link with DataServe.

[0073] Specifically, the standard nvme client establishes a link with the server through the following command:

[0074] nvme connect-t nettype-n"nqn.XX.yy.mm"-a IPAddr-s port;

[0075] The nvme client of the embodiment of the present application supports establishing links with multiple nodes at the same time. The command format is as follows:

[0076] nvme connect-t nettype-n"nqn.XX.yy.mm"-m MetaSvr:port;

[0077] "-m MetaSvr:port" specifies the address and port information of MetaServer, such as Figure 5 As shown. The client establishes a link with MetaServer through the connect command to obtain the current cluster information, including the storage server list, data partition information, address mapping relationship (or algorithm), etc. The client establishes NVMe-oF links to multiple storage service nodes based on the obtained storage server information. Each storage server needs to start the NVMe-oF Target service, register the same Target resource, and support request routing forwarding. When the storage server finds that the request address does not belong to this service, it can forward the request to the real destination server.

[0078] Furthermore, the access of business applications to the logical block device triggers the read and write process of the NVMe client, such as Figure 6 As shown, including:

[0079] 1) NVMeClient first receives a read / write request (such as Figure 6 The target address is obtained from the IO request X, IO request Y), and the logical address LBA and length Len information are obtained by parsing the NVMe request message;

[0080] 2) Based on the cluster composition information cached in maps, such as the cluster partition table, data server list, etc., the storage server DataSever where the LBA address is located is calculated through the hash algorithm, and the NVMe client object that is linked to the server is selected, and a request message of the NVMe-oF protocol type is assembled and sent to the target node, carrying information such as read and write LBA, Len, operation type, session number, etc. Figure 6 In the example, based on the maps mechanism, NVMe request X is sent to DataSever0 and NVMe request Y is sent to DataSever1.

[0081] 3) After the server completes reading and writing, it constructs a response message of the NVMe-oF protocol type and returns it to the NVMeClient client, such as Figure 5 NVMe response X, NVMe response Y in;

[0082] 4) The NVMeClient client reuses the same NVMe link ID to return the IO response result to the application layer, such as Figure 6 In the IO response X and IO response Y, the NVMe link ID belongs to the NVMe session ID created and used by the operating system layer, which is used to identify the NVMe link at the operating system level.

[0083] Furthermore, the metadata update process is as follows, including two update modes: active update and passive update. Specifically:

[0084] The client's MetaClient is responsible for reading the latest cluster metadata information from the cluster metadata server and saving it to the DataMap table. The user-state MetaClient module and the kernel-state NVMeCltService implement data interaction by sharing the DataMap. MetaClient supports active update and passive update. Active update means that the client obtains cluster metadata regularly, such as Figure 7 As shown, the following steps are included:

[0085] Step S701: The second virtual sub-client obtains metadata information from the metadata storage node at a regular interval;

[0086] Step S702: The second virtual sub-client obtains metadata information;

[0087] Step S703: The user state where the second virtual sub-client is located shares metadata information with the kernel state through a maps mechanism.

[0088] Passive update means: when the client NVMeCltService finds that the source address of the response message (equivalent to the response result in the above embodiment) does not match the target server of the request message, it triggers the metadata passive update process and notifies MetaClient to re-acquire the cluster metadata. Among them, DataSever supports forwarding requests whose target address is not its own to the correct server.

[0089] The specific process is as follows Figure 8 As shown, including:

[0090] Step S801: In kernel state, the source address of the response result does not match the target node of the IO read / write request, triggering a passive update of metadata;

[0091] Step S802: The second virtual sub-client obtains metadata information from the metadata storage node at the next moment of the current moment;

[0092] Step S803: The second virtual sub-client obtains metadata information;

[0093] Step S804: The user state where the second virtual sub-client is located shares metadata information with the kernel state through a maps mechanism.

[0094] Optionally, there is another alternative solution for the data plane link in the embodiment of the present application. Fig. 9 As shown, the client and the storage cluster only create one complete NVMe-oF link, and other links use a custom data transmission protocol. The client in the alternative solution also includes clients 911 and 912 (equivalent to the third virtual sub-client in the above embodiment), and transmits data through the underlying network channel (that is, it may not pass through the NVMe-oF Target). The client converts the NVMe request into a custom internal IO request and sends it to the target storage server. The internal IO response is encapsulated as a standard NVMe response and returned to the NVMeCltService service, that is, the client connects to an NVMe-oF Target server through an NVMe-oF link, but supports multiple underlying network channels to transmit data. The alternative solution helps to achieve higher IO performance.

[0095] In summary, the NVMe client command interface of the embodiment of the present application needs to specify the IP and port of the metadata server, and has obvious characteristics. In addition, the client of the embodiment of the present application establishes a link channel with the metadata server and the storage server in the storage cluster, and the network characteristics are very significant.

[0096] Furthermore, the embodiment of the present application expands the NVMe client to support multi-link concurrent access to the storage cluster, solving the performance bottleneck problem of the current distributed storage cluster single client, and solving the problem that the customized client cannot efficiently access the kernel IO driver. The transformation of the NVMe client using the kernel extension mechanism does not require direct modification of the kernel module, and the implementation and deployment of the solution are more flexible, solving the problem that the current implementation method needs to be bound to the kernel.

[0097] In this embodiment, a node access device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0098] Fig.10is a structural block diagram of a node access device according to an embodiment of the present application. Fig.10 As shown, the node access device includes:

[0099] An extension module 1002 is used to extend the first access client through a kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client;

[0100] The access module 1004 is configured to access the distributed storage nodes through the multiple first virtual sub-clients.

[0101] The embodiment of the present application, through the above steps, expands the first access client through the kernel extension technology to obtain the second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine the multiple first virtual sub-clients corresponding to the second access client; and access the distributed storage node through the multiple first virtual sub-clients. In other words, by expanding the first access client through the kernel extension technology, a second access client that can access the distributed storage node through multiple first virtual sub-clients is obtained, which solves the technical problem that the existing NVMe client cannot concurrently access multiple storage nodes of the distributed storage. The technical effect of concurrent access to the distributed storage nodes is achieved.

[0102] In an exemplary embodiment, the access module 1004 is also used to determine the second virtual sub-client corresponding to the second access client according to the user state program when the second access client also includes a user state program; and access the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node.

[0103] In an exemplary embodiment, the access module 1004 is further configured to set a timer for the second virtual sub-client; and obtain metadata information in the metadata storage node within the timing duration according to the timing duration of the timer.

[0104] In an exemplary embodiment, the access module 1004 is further used to respectively establish links between the multiple first virtual sub-clients and the distributed storage nodes, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

[0105] In an exemplary embodiment, the access module 1004 is also used to parse the read and write requests sent by the block device through the second access client to obtain the target address; determine the target node corresponding to the target address through an address mapping algorithm; and send the read and write requests to the target node, wherein the read and write requests are sent through the first virtual sub-client linked to the target node.

[0106] In an exemplary embodiment, the access module 1004 is further used to determine the hash value corresponding to the target address through the address mapping algorithm; determine the target node corresponding to the hash value according to metadata information, wherein the metadata information is used to indicate the correspondence between the hash value and the target node.

[0107] In an exemplary embodiment, the access module 1004 is also used to receive a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node, an actual node, wherein the actual node is the node to which the read / write request actually belongs when the read / write request does not belong to the target node; parsing the response result to obtain a response address of the response result; and when the response address is different from the target address, accessing a metadata storage node at the next moment of the current moment to obtain the latest metadata information in the metadata storage node.

[0108] In an exemplary embodiment, the extension module 1002 is also used for the multiple first virtual sub-clients to include at least one of the following: multiple first virtual sub-clients determined based on a first data transmission protocol; a first virtual sub-client determined based on the first data transmission protocol and multiple third virtual sub-clients determined based on a second data transmission protocol.

[0109] In an exemplary embodiment, the access module 1004 is also used to, when the virtual sub-client is a third virtual sub-client, pass the read and write request to the third virtual sub-client through the first virtual sub-client; convert the read and write request into a read and write request based on the second data transmission protocol through the third virtual sub-client, and send the read and write request based on the second data transmission protocol to the target node.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a readable storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0111] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0112] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0113] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0114] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0115] S1, expanding a first access client by kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client;

[0116] S2: Access the distributed storage node through the multiple first virtual sub-clients.

[0117] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0118] Optionally, in this embodiment, the above-mentioned electronic device can also be configured to execute the above-mentioned step S1 through a computer program, expand the first access client through kernel extension technology, and obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine multiple first virtual sub-clients corresponding to the second access client; S2, access the distributed storage node through the multiple first virtual sub-clients.

[0119] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0120] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0121] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A node access method, characterized in that: include: Expanding the first access client by kernel expansion technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client; The distributed storage node is accessed through the multiple first virtual sub-clients.

2. The node access method according to claim 1, characterized in that Before accessing the distributed storage nodes through the multiple first virtual sub-clients, the method further includes: In the case where the second access client further includes a user state program, determining a second virtual sub-client corresponding to the second access client according to the user state program; The metadata storage node is accessed through the second virtual sub-client to obtain metadata information in the metadata storage node.

3. The node access method according to claim 2, characterized in that , accessing the metadata storage node through the second virtual sub-client to obtain metadata information in the metadata storage node, including: Setting a timer for the second virtual sub-client; The metadata information in the metadata storage node within the timing duration is acquired according to the timing duration of the timer.

4. The node access method according to claim 1, characterized in that Before accessing the distributed storage nodes through the multiple first virtual sub-clients, the method further includes: Establishing links between the plurality of first virtual sub-clients and the distributed storage node respectively, wherein the links are established through metadata information, wherein the metadata information is obtained by the second access client from the metadata storage node.

5. The node access method according to claim 1, characterized in that: Accessing the distributed storage nodes through the multiple first virtual sub-clients includes: Parsing the read and write request sent by the block device through the second access client to obtain a target address; Determine the target node corresponding to the target address by an address mapping algorithm; The read / write request is sent to the target node, wherein the read / write request is sent through a first virtual sub-client linked to the target node.

6. The node access method according to claim 5, characterized in that After sending the read / write request to the target node, the method further includes: Receive a response result fed back by a response node, wherein the response result is a response result fed back by the response node based on the read / write request, and the response node includes one of the following: a target node and an actual node, wherein the actual node is a node to which the read / write request actually belongs when the read / write request does not belong to the target node; Parsing the response result to obtain a response address of the response result; In the case that the response address is different from the target address, the metadata storage node is accessed at a time next to the current time to obtain the latest metadata information in the metadata storage node.

7. The node access method according to claim 5, characterized in that ,include: The plurality of first virtual sub-clients include at least one of the following: a plurality of first virtual sub-clients determined based on a first data transmission protocol; A first virtual sub-client determined based on the first data transmission protocol and a plurality of third virtual sub-clients determined based on the second data transmission protocol.

8. The node access method according to claim 7, characterized in that , sending the read / write request to the target node, including: In the case where the virtual sub-client is a third virtual sub-client, the read / write request is transmitted to the third virtual sub-client through the first virtual sub-client; The read / write request is converted into a read / write request based on the second data transmission protocol through the third virtual sub-client, and the read / write request based on the second data transmission protocol is sent to the target node.

9. A node access device, characterized in that: include: An extension module, used to extend the first access client through kernel extension technology to obtain a second access client, wherein the second access client includes a kernel state program, and the kernel state program is used to determine a plurality of first virtual sub-clients corresponding to the second access client; An access module is used to access the distributed storage nodes through the multiple first virtual sub-clients.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images