SPDK storage gateway client request transparent transmission method based on client identifier

Abstract

The invention relates to an SPDK storage gateway client request unvarnished transmission method based on a client identifier, and the method comprises the steps: adding an SPDK storage gateway in an own distributed storage system, converting a link identifier-based storage lock of a standard SCSI or NVMe protocol into a client identifier-based storage lock through employing a PBDX driver, and transmitting the client identifier-based storage lock to the SPDK storage gateway. Compared with the prior art, the method has the advantages that stateless storage of the storage gateway is realized, dependence on an open source synchronization component is effectively reduced, deployment of the storage gateway is simplified, and the storage access client is marked by the client identifier, and the client request is transmitted to the bottom distributed storage system at the SPDK storage gateway layer. The fault risk points of the whole storage system are reduced, meanwhile, multiple clients are supported to share and access the same storage volume in a cross-protocol mode, the compatibility of different operating system platforms is improved, and the flexibility and the utilization rate of storage resources are improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed storage gateway client request transparent transmission technology based on a unique client identifier and implemented by using SPDK (Storage Performance Development Kit). Background Art

[0002] At present, most of the open source or commercial implementations of distributed storage are developed based on the Linux operating system platform. For storage access requirements of other non-Linux platforms (such as Windows, ESX, etc.), it is necessary to build a storage gateway based on the standard SCSI (Small Computer System Interface) / NVMe (Non-Volatile Memory Express) protocol.

[0003] In the application scenario where multiple hosts share the same storage resource, it is inevitable to use two storage lock mechanisms, Persistent Reservation (PR) and Compare-And-Write (CAW), to implement access rights control for different storage access clients. The PR and CAW lock mechanisms in the standard SCSI / NVMe protocol are based on one-to-one link information access rights control, and their access rights control is implemented at the SCSI / NVMe Target layer of the storage gateway. For the multi-path access method of the client, it is necessary to synchronize the PR storage lock information between the storage gateways. At the same time, in order to ensure the strict atomicity of the CAW lock, it is necessary to select the master gateway node for processing the CAW lock between the storage gateways. If other slave gateway nodes receive a CAW request, they will forward the request to the master gateway node for processing. Whether it is the storage gateway synchronization of the PR storage lock or the master selection and forwarding logic of the CAW storage lock, they all rely on complex cluster kits or distributed component implementations (such as DLM, Corosync, etcd, etc.). This not only further increases the complexity of the self-owned distributed storage system, but also increases the risk points of the entire storage system, bringing certain difficulties to the operation and maintenance of distributed storage. Summary of the invention

[0004] The purpose of the present invention is to provide a SPDK storage gateway client request transparent transmission method based on client identification, aiming to improve the compatibility of the proprietary distributed storage system with different operating system platforms, realize access permission control and support multiple clients to share access to the same storage volume across protocols, while simplifying deployment and reducing the complexity of the proprietary distributed storage system and improving the performance of distributed storage locks.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for transparently transmitting a SPDK storage gateway client request based on a client identifier, the method comprising:

[0007] Add SPDK storage gateway to the company's distributed storage system;

[0008] Establish a connection with the storage access client through the Target layer of the SPDK storage gateway and set the client identifier for the client;

[0009] Get the client request and determine the request type;

[0010] If the client request is a PR lock request, the client request with the client identifier is passed to the PBDX driver for conversion using the PR transparent transmission instruction and transparently transmitted to the own distributed storage system for processing;

[0011] If the client request is an IO request and the client identifier has been set, the client identifier is applied to each IO request descriptor, and then the IO request with the client identifier is converted by the PBDX driver and transparently transmitted to the own distributed storage system for processing;

[0012] If the client request is an IO request and the client identifier is not set, the IO operation is performed directly in the own distributed storage system without the need for access permission check;

[0013] If the client request is a query request, the corresponding storage access client information is obtained from the own distributed storage system according to the query request, and the corresponding storage access client is found out according to the client identifier in the information.

[0014] After processing, operation and reverse query are completed, the results are returned to the client.

[0015] Furthermore, the PBDX driver has a built-in command parser, and the BDEV layer of the SPDK storage gateway has a built-in distributed storage system access client driver for converting client requests of the standard protocol into corresponding requests within the distributed storage system.

[0016] Furthermore, the client identifier of any client is unique and unchanged.

[0017] Furthermore, the client identifier is set by the SPDK Target layer according to the configuration of the storage access client and saved in the corresponding storage access client connection description structure.

[0018] Furthermore, the setting of the client identifier includes:

[0019] If the client accesses its own distributed storage system through iSCSI (Internet Small Computer System Interface), the client identifier is set according to the IQN (iSCSI Qualified Name) of the iSCSI client;

[0020] If the client accesses its own distributed storage system via NVMe (Non-Volatile Memory express, non-volatile memory host controller interface specification), the client identifier is set according to the NQN (NVMe Qualified Name) of the NVMe client;

[0021] If the client accesses its own distributed storage system through VHOST-SCSI (Virtual Host SCSI), the client ID is set according to the vhost controller (virtual host controller).

[0022] Furthermore, when the acquired request is a PR lock request, the client identifier of the client issuing the request is set for the PR transparent transmission instruction.

[0023] Furthermore, if the PR transparent instruction client identifier is not set, it is regarded as a parameter error and EINVAL is returned.

[0024] Furthermore, the processing of the self-owned distributed storage system includes:

[0025] receiving a client request with a client identifier;

[0026] Identify the client ID and grant the client corresponding permissions based on the client ID code;

[0027] Verify whether the client's permissions allow the client to perform the operations requested by the client. If the verification is successful, perform the corresponding operations according to the request and return the operation results to the client. If the verification fails, reject the client request.

[0028] Furthermore, by calling the BDEV layer IO interface of the SPDK storage gateway, the IO request with the client identifier is handed over to the PBDX driver for conversion and transparently transmitted to the own distributed storage system for processing.

[0029] Furthermore, the IO interface includes read, write, COMPARE and WRITE ZERO interfaces.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention provides a method for transparent transmission of SPDK storage gateway client requests based on client identifiers, converts a storage lock based on a link identifier of a standard SCSI or NVMe protocol into a storage lock based on a client identifier, implements access control and supports multiple clients to share access to the same storage volume across protocols, improves compatibility with different operating system platforms and the flexibility and utilization of storage resources, transparently transmits client requests to a proprietary distributed storage system at the SPDK storage gateway layer, implements statelessness of the storage gateway, effectively reduces dependence on open source synchronization components, simplifies storage gateway deployment, and reduces the risk of failures in the entire storage system;

[0032] The PBDX driver comes with a command parser that can directly parse and convert client requests of SCSI or NVMe standard protocols, avoiding multiple protocol conversions and major modifications to the SPDK BDEV interface layer, and ensuring the integrity of the request information;

[0033] By directly transmitting client requests to its own distributed storage system for processing, the processing time at the storage gateway layer is reduced and the efficiency of storage access is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the overall framework diagram of the SPDK storage gateway client request processing of the present invention;

[0035] Figure 2 This is a flow chart of the client request interaction processing within the SPDK storage gateway of the present invention. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0037] This embodiment discloses a method for transparently transmitting an SPDK storage gateway client request based on a client identifier.

[0038] In order to reduce the complexity of operation and maintenance and reduce component dependence, SPDK is selected as the basic component of the storage access gateway of the distributed storage system based on the distributed storage lock compatible with SCSIPR / CAW in the proprietary distributed storage system. By utilizing the full user state characteristics of SPDK, the proprietary distributed storage system access client driver PBDX is built into the SPDK BDEV layer, and the link-based gateway layer storage lock is converted into a client identifier (TAG)-based storage lock. The client request is transmitted to the proprietary distributed storage system for processing through PBDX.

[0039] In this embodiment, the proprietary distributed storage system is an IOS storage cluster.

[0040] The specific implementation process of the SPDK storage gateway transparent transmission method based on client identification is as follows:

[0041] First, add an SPDK storage gateway to the distributed storage cluster.

[0042] The IOS storage cluster stores data on multiple independent devices in a dispersed manner, which together form a virtual storage resource pool through network connections. Therefore, multiple SPDK storage gateway nodes are added to form an SPDK storage kingdom cluster. Various storage access clients such as NVMe, iSCSI, etc. connect to multiple SPDK storage gateway nodes through multipathing.

[0043] Subsequently, the system establishes a connection with the storage access client through the Target layer of the SPDK storage gateway and sets a client identifier (TAG) for the client;

[0044] The client identifier is a unique client identifier parameter used to set different access permissions based on the client. The client identifier ensures that the permissions are consistent when the same client accesses the IOS storage cluster through different SPDK storage gateways. When the client identifier is not set, it is considered that no access permission control check is required, that is, in the storage system compatible upgrade scenario, the old version of the client identifier is not configured, and normal read and write access to the IOS storage cluster can be performed after the upgrade.

[0045] The client ID is set as follows:

[0046] If the client accesses its own distributed storage system through iSCSI (Internet Small Computer System Interface), the client identifier is set according to the IQN (iSCSI Qualified Name) of the iSCSI client;

[0047] If the client accesses its own distributed storage system via NVMe (Non-Volatile Memory express, non-volatile memory host controller interface specification), the client identifier is set according to the NQN (NVMe Qualified Name) of the NVMe client;

[0048] If the client accesses its own distributed storage system through VHOST-SCSI (Virtual Host SCSI), the client ID is set according to the vhost controller (virtual host controller). Different clients use different virtual machines and have different vhost controllers.

[0049] After setting the client identifier, the Target layer of the SPDK storage gateway obtains the client request through the connection with the client, and then determines the request type based on the specific content of the client request.

[0050] The judgment situations include: the client request is a PR lock request, the client request is an IO request and the client identifier is set, the client request is an IO request and the client identifier is not set, and the client request is a query request.

[0051] Based on the judgment, the SPDK storage gateway processes the client request as follows:

[0052] If the client request is a PR lock request, the client request with the client identifier is passed to the PBDX driver for conversion using the PR transparent transmission instruction and transparently transmitted to the own distributed storage system for processing;

[0053] If the client request is an IO request and the client identifier has been set, the client identifier is applied to each IO request descriptor, and then the IO request with the client identifier is converted by the PBDX driver and transparently transmitted to the own distributed storage system for processing;

[0054] If the client request is an IO request and the client identifier is not set, the IO operation is performed directly in the own distributed storage system without the need for access permission check;

[0055] If the client request is a query request, then obtain the information of the corresponding storage access client from the own distributed storage system according to the query request, and find out the corresponding storage access client according to the client identifier in the information;

[0056] In the above operation, before using the PR transparent transmission instruction, the client identifier of the PR transparent transmission instruction must be assigned. The specific value assigned is the same as the client identifier of the client that issued the request. If the PR transparent transmission instruction client identifier is 0, it is considered a parameter error and EINVAL is returned.

[0057] Client requests that are passed through to the IOS storage cluster for processing will go through the following operations:

[0058] The system identifies the client ID and grants the client corresponding permissions according to the configured access permissions.

[0059] Verify whether the client's permissions allow the client to perform the operations requested by the client. If the verification is successful, perform the corresponding operations according to the request and return the operation results to the client. If the verification fails, reject the client request.

[0060] Finally, the IOS storage cluster returns the final processing result to the client.

[0061] The IO interface includes read, write, COMPARE and WRITE ZERO interfaces.

[0062] After the IOS storage cluster receives a client request with a client ID, the system identifies the client ID and grants the client corresponding permissions based on the client ID code;

[0063] Verify whether the client's permissions allow the client to perform the operations requested by the client. If the verification is successful, perform the corresponding operations according to the request and return the operation results to the client. If the verification fails, reject the client request.

[0064] Figure 1 This is the overall framework diagram of the SPDK storage gateway client request processing of the present invention. Various storage access clients such as NVMe, iSCSI, etc. connect to multiple SPDK storage gateway nodes through multi-path mode, and the SPDK Target layer sets a unique client identification TAG according to the storage client NQN / IQN / vhost controller. The identification TAG of the same storage access client on different storage gateway nodes is the same, and different storage access clients set different identification TAGs. The PR lock request and IO request sent to the underlying distributed block storage through the SPDK storage gateway layer are all with the client identification TAG.

[0065] Therefore, the IO request client identifier of the same client reaching the backend IOS storage cluster via different storage gateways is the same, and the client identifiers of read and write requests for the same storage volume from different clients are different even if they reach the backend IOS storage cluster via the same storage gateway. This allows the entire set of original standard protocol storage lock logic to be fully transparently transmitted to the backend IOS storage cluster for implementation, without the need to synchronize lock information between the various nodes of the storage gateway.

[0066] Figure 2 The flowchart of the client request interaction process in the SPDK storage gateway of the present invention is shown in FIG. PBDX is the driver in the SPDK BDEV layer involved in the present invention. Target is responsible for configuring the client identification TAG of the storage access client, and when receiving the IO request and the PR lock request, it sets the identification TAG of the corresponding client for the IO request, transparently transmits the client request to the PBDX driver for processing, and the Target interacts with the PBDX driver through the BDEV interface.

[0067] The PBDX driver has built-in standard NVMe protocol command and SCSI protocol command parsers, which can parse the raw NVMe commands / SCSI CDB commands transmitted by the upper-layer NVMFTarget / SCSI Target through the BDEV PR transparent interface. PBDX converts the PR lock request and read / write request into a distributed storage PDS internal command (PDS cmd) and sends it to the backend IOS storage cluster for processing.

[0068] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for transparent transmission of SPDK storage gateway client requests based on client identification, characterized in that: The method comprises: Add SPDK storage gateway to the company's distributed storage system; Establish a connection with the storage access client through the Target layer of the SPDK storage gateway and set the client identifier for the client; Get the client request and determine the request type; If the client request is a PR lock request, the lock request is passed to the PBDX driver using the PR transparent transmission instruction for conversion and transparent transmission to the own distributed storage system for processing; If the client request is an IO request and the client identifier has been set, the client identifier is applied to each IO request descriptor, and then the IO request with the client identifier is converted by the PBDX driver and transparently transmitted to the own distributed storage system for processing; If the client request is an IO request and the client identifier is not set, the IO operation is performed directly in the own distributed storage system; If the client request is a query request, the information of the corresponding storage access client is obtained from the own distributed storage system according to the query request, and the corresponding storage access client is found out according to the client identifier in the information; After processing, operation and reverse query are completed, the results are returned to the client.

2. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: The PBDX driver has a built-in command parser and is a built-in distributed storage system access client driver in the BDEV layer of the SPDK storage gateway, which is used to convert client requests of the standard protocol into corresponding requests within the distributed storage system.

3. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: The client ID of any client is unique and unchanged.

4. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: The client identifier is set by the SPDK Target layer according to the configuration of the storage access client and saved in the corresponding storage access client connection description structure.

5. According to the method of claim 4, a method for transparent transmission of SPDK storage gateway client requests based on client identification is characterized in that: The setting of the client identifier includes: If the client accesses its own distributed storage system through iSCSI, set the client ID according to the IQN of the iSCSI client; If the client accesses its own distributed storage system via NVMe, the client ID is set according to the NQN of the NVMe client; If the client accesses its own distributed storage system through VHOST-SCSI, the client ID is set according to the vhost controller.

6. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: When the acquired request is a PR lock request, the corresponding client identifier is acquired according to the client information, and the client identifier of the client issuing the request is set for the PR lock instruction.

7. According to the method of claim 6, a method for transparent transmission of SPDK storage gateway client requests based on client identification, characterized in that: If the client identifier of the client of the PR instruction cannot be obtained, it is considered as a parameter error and EINVAL is returned.

8. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: The processing of the self-owned distributed storage system includes: receiving a client request with a client identifier; Identify the client ID and grant the client corresponding permissions based on the client ID code; Verify whether the client's permissions allow the client to perform the operations requested by the client. If the verification is successful, perform the corresponding operations according to the request and return the operation results to the client. If the verification fails, reject the client request.

9. According to the method of claim 1, a client identification-based SPDK storage gateway client request transparent transmission method is characterized in that: By calling the BDEV layer IO interface of the SPDK storage gateway, the IO request with the client identifier is converted by the PBDX driver and transmitted to the company's own distributed storage system for processing.

10. A method for transparent transmission of SPDK storage gateway client requests based on client identification according to claim 9, characterized in that: The IO interface includes read, write, COMPARE and WRITE ZERO interfaces.

Images