Storage cluster data processing method, apparatus and system

Abstract

This application provides a storage cluster data processing method, apparatus, and system, applied in the field of information processing technology. The storage cluster data processing method includes splitting target data used to access the storage cluster; and enabling a target host to access the target storage in the storage cluster via a target node based on a cross-node valid namespace identifier located by a global namespace identifier obtained from the split target data. By defining a custom NVM command set with a cross-node valid global namespace identifier in the general NVM command set format, the cross-node valid namespace identifier can be located using the global namespace. After splitting and forwarding the new custom NVM command set, global access to the storage cluster by the host can be achieved.

Description

Technical Field

[0001] This application relates to the field of information processing technology, specifically to a storage cluster data processing method, apparatus, and system. Background Technology

[0002] NVMe-oF (NVMe over Fabrics) is a specification that connects a host to NVMe solid-state storage via network nodes. It can use Fabrics technologies such as RDMA (Remote Direct Memory Access) or Fibre Channel (FC) architecture to replace PCIe (peripheral component interconnect express, a high-speed serial computer expansion bus standard) for data transmission.

[0003] While the NVM (Non-Volatile Memory Express) command set universal format allows for point-to-point data transfer from client hosts to gateway nodes, with no data flow between gateway nodes, greatly enhancing flexibility and scalability. This enables the low latency and high concurrency characteristics of NVMe (Non-Volatile Memory Express, a structure designed specifically for flash-based storage devices) to be extended from the server level to the entire data center level. However, precisely because data transfer is point-to-point, the overall performance of NVMe-based storage clusters (such as those with flash clusters and flash expansion enclosures) cannot be fully utilized. Summary of the Invention

[0004] This specification provides a storage cluster data processing method, apparatus, and system. Based on the NVM command format, a custom NVM command set with a cross-node valid global namespace identifier (also known as a global namespace field) is defined. The global namespace identifier is used to locate the cross-node valid namespace identifier (also known as a namespace field). After processing such as splitting and forwarding new commands, the host can perform global access to the storage cluster.

[0005] The embodiments in this specification provide the following technical solutions:

[0006] This specification provides an embodiment of a storage cluster data processing method, including:

[0007] The target data used to access the storage cluster is split, wherein the target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format, the global namespace identifier being used to locate a valid namespace identifier across nodes;

[0008] Based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data, the target host accesses the target storage in the storage cluster through the target node, wherein the target node is the gateway node in the storage gateway cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier.

[0009] This specification also provides a storage cluster data processing method, deployed in a storage gateway cluster, comprising:

[0010] The target data is received by the target host through the first gateway node, wherein the first gateway node is a gateway node in the storage gateway cluster that has a communication connection with the target host. The target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format. The global namespace identifier is used to locate the namespace identifier that is valid across nodes.

[0011] The first gateway node splits the target data for accessing the storage cluster, and based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the split target data, the target host accesses the target storage in the storage cluster through the target node, wherein the target node is the gateway node in the storage gateway cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier.

[0012] Optionally, the storage cluster data processing method further includes:

[0013] Extend a new gateway node into a storage gateway cluster, wherein the new gateway node is configured to perform the following operations: obtain the cluster configuration of the storage gateway cluster and initialize the cluster configuration; after completing the configuration, perform metadata reconstruction processing, wherein the new gateway node shares some metadata from several other gateway nodes.

[0014] Optionally, the new gateway node sequentially performs migration operations on the metadata that needs to be migrated in the other gateway nodes through a metadata atomic query command. The metadata atomic query command is a metadata atomic query command defined based on the general format of the NVM command set contained in the target instruction. The metadata atomic query command is configured with a lock mode, a starting logical block address, and a number of logical blocks. The lock mode is used for locking services when performing atomic query operations on metadata. The starting logical block address is used to represent the starting address when performing atomic query operations on metadata. The number of logical blocks is used to represent the number of logical blocks that need to be atomically queried starting from the starting logical block address when performing atomic query operations on metadata.

[0015] Optionally, in the atomic query operation of metadata on the storage cluster based on the metadata atomic query command, the storage cluster data processing method further includes:

[0016] Determine whether the metadata has been migrated to the new gateway node;

[0017] If so, the target storage in the storage cluster is accessed through the new gateway node;

[0018] If not, the lock mode is set to the locking mode, and the metadata atomic query command is sent to other nodes containing the metadata. After the other nodes containing the metadata lock the metadata, the target storage in the storage cluster is accessed through the new gateway node to obtain the metadata. After obtaining the metadata, the lock mode is set to the lock release mode so that the other nodes containing the metadata release the lock on the metadata.

[0019] Optionally, the storage cluster data processing method further includes: setting a global namespace service in the storage gateway cluster, and filling in the gateway node ID and the global namespace identifier in the global namespace service according to the mapping relationship between the gateway node ID and the cross-node valid namespace identifier located by the global namespace identifier.

[0020] Optionally, the storage cluster data processing method further includes: setting up several local namespace services in each gateway node of the storage gateway cluster, wherein the total capacity of the several local namespace services is the capacity of the global namespace service.

[0021] This specification also provides a storage cluster data processing apparatus, comprising:

[0022] The splitting module is used to split the target data accessing the storage cluster. The target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format. The global namespace identifier is used to locate the valid namespace identifier across nodes.

[0023] The forwarding module is used to enable the target host to access the target storage in the storage cluster through the target node based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data. The target node is the gateway node in the storage gateway cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier.

[0024] This specification also provides a storage cluster data processing system, including: a target host, a storage gateway cluster, and a storage cluster, wherein the target host is communicatively connected to at least one gateway node in the storage gateway cluster, each gateway node in the storage gateway cluster is communicatively connected to the storage cluster, and the target host, the storage gateway cluster, or the storage cluster are deployed with the storage cluster data processing method described in any embodiment of this specification.

[0025] This specification also provides a storage cluster data processing system, including: a request forwarding node, a target host, a storage gateway cluster, and a storage cluster, wherein the target host is communicatively connected to the request forwarding node, the request forwarding node is communicatively connected to at least one gateway node in the storage gateway cluster, each gateway node in the storage gateway cluster is communicatively connected to the storage cluster, and the request forwarding node is deployed with the storage cluster data processing method described in any embodiment of this specification.

[0026] Compared with the prior art, the beneficial effects that can be achieved by the above-mentioned at least one technical solution adopted in the embodiments of this specification include at least the following: by customizing the NVMe command set on the general format of the NVM command set, the NVMe command set is expanded, so that the scope of the namespace accessible by the custom NVM command set is expanded from a single node to cross-node, thereby realizing global access of the storage cluster by the client host.

[0027] After deploying a storage gateway cluster, client hosts can globally read and write the global namespace by connecting to any gateway node in the cluster. The cluster also enables load balancing of data processing, improving the throughput of the storage system. Furthermore, by connecting to multiple gateway nodes, high availability and load balancing can be achieved. For example, if one gateway node is disconnected, the others can still function, thus ensuring high availability.

[0028] After deployment to the client host, the flexibility to deploy I / O splitting and I / O forwarding functions to the client host can be retained, and data splitting and forwarding can be implemented directly on this host. This can reduce the traffic forwarded within the storage gateway cluster. This deployment method can implement the multi-path function of the NVMe layer on the client host. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a structural diagram of the general format of the NVM command set;

[0031] Figure 2 This is a flowchart of how a client host accesses storage through a gateway node within a valid namespace within a node.

[0032] Figure 3 This is a schematic diagram of the structure of a custom NVM command set;

[0033] Figure 4 This is a flowchart of data processing based on a custom NVM command set;

[0034] Figure 5 This is a flowchart of the gateway node splitting and forwarding process after deployment in a storage gateway cluster;

[0035] Figure 6 This is a flowchart of the process of splitting the gateway node into multiple sub-commands and forwarding them after deployment in the storage gateway cluster;

[0036] Figure 7 This is a schematic diagram of the structure of a storage gateway cluster providing global namespace services;

[0037] Figure 8 This is a schematic diagram of the structure of a storage gateway cluster that provides global namespace services based on local namespace services;

[0038] Figure 9This is a schematic diagram of a structure that includes atomic query commands for metadata;

[0039] Figure 10 This is a schematic diagram of the expansion of the storage gateway cluster gateway nodes;

[0040] Figure 11 This is a flowchart illustrating the data processing based on atomic query commands when metadata is hit in the extended gateway node;

[0041] Figure 12 This is a flowchart illustrating data processing based on atomic query commands when metadata is not hit in the extended gateway node;

[0042] Figure 13 This is a flowchart illustrating how metadata is partially processed based on atomic query commands when it is hit in the extended gateway node. Detailed Implementation

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0046] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0048] like Figure 1 As shown, the general format of the NVM command set defines a command identifier, an opcode, a namespace identifier (namespace ID), a metadata pointer, a data pointer, a command Dword (Command Dword 10 to Command Dword 15 shown in the figure), and unused reserved fields. The NVM command set can be used to perform corresponding data processing operations.

[0049] like Figure 2 As shown, in the end-to-end access method from the client host to the storage cluster, the data flow is usually sent from the client host to the gateway node first. Then, the NVMe target in the gateway node finds the mapped NVMe solid-state storage device (denoted as NVMe SSD) in the gateway node according to the namespace identifier (NamespaceID). The gateway node then accesses the corresponding solid-state storage (SSD) in the NVMe solid-state storage cluster (denoted as NVMe SSDs) through the backend network via the NVMe initiator.

[0050] Therefore, based on the existing NVM command set's common format, client hosts can achieve point-to-point transmission to gateway nodes, resulting in no data flow between gateway nodes. However, hosts cannot access and process data from other solid-state drives (SSDs) from their connected gateway nodes from a global perspective, thus failing to leverage the overall performance of the NVMe storage cluster.

[0051] In light of this, through in-depth research and improvement exploration of the NVM command set and its format, as well as NVMe storage clusters and their data access methods, it was discovered that NVMe data commands already contain the command object—the namespace identifier (i.e., Namespace ID), as mentioned above. Figure 1The example shown is the Namespace ID in the general NVM command set format. However, the valid range of this Namespace ID is only valid within one NVM subsystem within a gateway node. Additionally, some fields in the general NVM command set format remain unused, such as those mentioned above. Figure 1 The Reserved field is shown.

[0052] Based on this, the embodiments of this specification propose a data processing scheme for a storage cluster:

[0053] like Figure 3 As shown, by utilizing the namespace identifier and reserved fields in the NVM command set and its format, a custom NVM command set format with globally valid namespace identifiers can be created by inserting a globally valid namespace identifier (Global NamespaceIdentifier) ​​into a custom NVM command. The global namespace identifier is used to locate the namespace identifier that is valid across nodes. That is, the global namespace identifier replaces the namespace identifier (Namespace Identifier, i.e., Namespace ID) that is only valid in the NVM subsystem in the original general format, so that the namespace identifier in the custom NVM command set can be globally valid across gateway nodes.

[0054] like Figure 4 As shown, based on a custom NVM command set containing a global namespace identifier, the following steps can be executed when a host accesses data from storage storage within a storage cluster via a storage gateway cluster:

[0055] Step S202: Split the target data used to access the storage cluster, wherein the target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format, and the global namespace identifier is used to locate the namespace identifier valid across nodes.

[0056] Step S204: Based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data, the target host accesses the target storage in the storage cluster through the target node. The target node is the gateway node in the storage gateway cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier. Therefore, according to the mapping relationship between the namespace identifier and the gateway node, the data access instruction corresponding to the namespace identifier is forwarded to the corresponding gateway node, thereby accessing the storage in the NVM subsystem's storage cluster through the gateway node.

[0057] By customizing the NVM command set based on the general NVM command set format, and utilizing the namespace identifier and reserved fields in the general NVM command set format, a globally valid global namespace identifier for the storage cluster is inserted into the custom NVM command set. By splitting and forwarding the cross-node valid namespace identifier located by the global namespace identifier, the host can perform global access processing of the storage cluster based on a single transmission path after the custom NVM command set is implemented.

[0058] In one implementation, auxiliary identifiers can be defined using reserved fields in a custom NVM command set format, which can help with the global access processing.

[0059] In one example, such as Figure 3 As shown, a custom Server NodeIdentifier can be defined to identify the gateway node ID and to track which gateway node the NVMe command originates from. This allows for identification and tracking when the host performs global access processing on the storage cluster, thereby improving data processing efficiency.

[0060] In one example, such as Figure 3 As shown, a custom magic number field can be used to validate the global namespace identifier in the target data, preventing confusion between the global namespace identifier and other custom Vendor commands with the same opcode. It should be noted that the magic number setting can be determined according to actual needs and is not limited here.

[0061] In some implementations, processing can be split and forwarded within an execution entity (such as a host, front-end network, storage gateway cluster (or gateway node), back-end network, storage cluster, etc.), thereby enabling the host to perform global access processing on the storage devices in the storage cluster.

[0062] In implementation, when a host accesses data from the storage cluster's memory through the storage gateway cluster based on a custom NVM command set containing a global namespace identifier, the aforementioned... Figure 4 The data processing method shown can deploy the splitting and forwarding steps to the gateway node. Besides this method, other alternative deployment schemes include:

[0063] 1) Deployed to a separate forwarding node, which is responsible for receiving requests from client hosts and forwarding them to the corresponding gateway node;

[0064] 2) Deploying to a virtual machine within the client host, although it consumes the virtual machine's computing power resources on the client host, can save hardware resources on the gateway node;

[0065] 3) Deploying to the client host OS (operating system) consumes the computing resources of the client host operating system, but can save the hardware resources of the gateway node;

[0066] 4) Deploying to a container on the client host consumes computing power from the client host, but saves hardware resources on the gateway node.

[0067] 5) Deploying smart network cards to client hosts. Although smart network cards have lower hardware resources and performance than gateway nodes, they offload the computing power of the original gateway nodes, saving computing power for both the gateway nodes and client hosts.

[0068] Therefore, all of the above deployment schemes can enable the host to perform global data access and processing on the storage cluster through the storage gateway cluster, and can also give full play to the overall performance of data processing of the NVMe-based storage cluster.

[0069] The following example illustrates the deployment of splitting and forwarding in a gateway node. It should be noted that this example is based on the deployment of splitting and forwarding in a gateway node; those skilled in the art should understand that the deployment of splitting and / or forwarding in other entities can be done by referring to the following illustrations, and will not be elaborated upon further.

[0070] In some implementations, two new functional modules are added to the gateway node: a command splitting module and a command forwarding module. The command splitting module can split the target data after the NVMe target receives a request from the client host. The command forwarding module can forward the split sub-data to the corresponding gateway node according to the splitting result. Thus, the relevant gateway node establishes a data access and processing service channel with the storage cluster, enabling the host to access the storage cluster globally based on a single transmission channel.

[0071] In one example, such as Figure 5 As shown, the client host sends an NVMe command to gateway node 2 in the storage gateway cluster. Gateway node 2 then populates the custom NVMe command with its gateway node ID and global namespace ID, and forwards the command to gateway node 1. Gateway node 1 parses the global namespace ID, finds its own namespace ID, and sends the NVMe command to the corresponding storage device in the backend storage cluster via the NVMe initiator. Correspondingly, the storage device returns data processing service information to gateway node 1, which then returns the service information to gateway node 2, which in turn returns the service information to the client host.

[0072] In one example, such as Figure 6 As shown, the client host sends an NVMe I / O command to gateway node 2 in the storage gateway cluster. Gateway node 2's I / O command splitting module splits the I / O command into three NVMe I / O sub-commands. The I / O command forwarding module sends the different sub-commands obtained from the custom NVMe command to different gateway nodes; for example, the first sub-command is sent to gateway node 1, the third sub-command to gateway node 3, while the NVMe I / O sub-commands that belong to gateway node 2's local service are served by gateway node 2 itself. Finally, after each gateway node completes its service for the I / O sub-command, it returns service information to gateway node 2. Gateway node 2 waits for all I / O sub-commands to complete their service before returning the final NVMe I / O completion status to the client host.

[0073] Based on a custom NVM command set containing a global namespace identifier, and by splitting the custom command data and forwarding subcommands between node clusters, subcommands are serviced by the corresponding nodes. Specifically, a command splitting module and a command forwarding module can be added to the gateway node. The command splitting module can extract the valid Namespace ID in the corresponding NVM subsystem from the global namespace identifier, and the command forwarding module can access the data according to the gateway node corresponding to the Namespace ID. This allows any client host to call the entire storage gateway cluster through a single transmission path. This not only realizes the functions of data distribution and clustering, but also makes full use of the performance and redundancy space of the gateway node, and better enables the host to perform global data access processing on the storage cluster.

[0074] In some implementations, given that each additional NVMe Target in a cluster without forwarding functionality requires an additional "N-1" connections, a global namespace service can be set up in the storage gateway cluster. This allows the data access service to be directly performed on the storage cluster's memory based on the set global namespace service when a host makes a request to the storage gateway cluster.

[0075] In one example, such as Figure 7 As shown, corresponding global namespaces, such as Global Namespace0 and Global Namespace1, are set up in the storage gateway cluster to form a symmetrical global namespace service. In this case, NVMe commands are served in the same way in each gateway node, and each gateway node can provide services through this global namespace, and can provide services for Global Namespace0 and Global Namespace1.

[0076] In one example, such as Figure 8 As shown, local namespaces are set up on each gateway node in the storage gateway cluster. For example, local namespaces such as Local Namespace0 and Local Namespace1 are set up on gateway node 1 to gateway node X. That is, each gateway node provides a portion of the space required by the namespace, and the total capacity of the local namespaces on all gateway nodes is equal to the capacity of the global namespace.

[0077] After NVMe I / O commands are issued from the host to the gateway node, the I / O subcommands corresponding to the split namespaces can be forwarded in the following three scenarios:

[0078] a) The logical block address (LBA) range of NVMe I / O commands is all within this gateway node, so the commands can be served on this node;

[0079] b) The LBA scope of NVMe I / O commands is all in other gateway nodes, and the commands are then forwarded to other gateway nodes for service.

[0080] c) The LBA scope of NVMe I / O commands is partly on the local node and partly on other gateway nodes. Within the local node, I / O commands are split into local I / O subcommands and forwarded I / O subcommands, and then the commands are sent to the corresponding gateway nodes for service.

[0081] Therefore, from the perspective of the client host, sending NVMe commands to any gateway node allows for reading and writing of the entire Global Namespace.

[0082] By setting up a global namespace service in the cluster, when I / O forwarding uses connection-oriented transmission, it is possible to save software and hardware resources for connections. Multiple NVMe Targets can share the connection of the forwarding module. For example, when using NVMe over RDMA or NVMe over TCP connection transmission, if the number of cluster nodes is N, only “N-1” connections are needed on each gateway node to achieve full network coverage (Full Mesh).

[0083] In some implementations, based on the aforementioned custom NVM command set, a custom NVM command set for performing metadata atomic query operations can be formed by customizing some command words in the general format of the NVM command set.

[0084] like Figure 9 As shown, a custom NVMe Vendor metadata atomic query command format is defined within the original NVM command set general format, specifically as follows:

[0085] 1) Set the Global Namespace Identifier to locate a valid Namespace ID across nodes. The Global Namespace Identifier is a required setting.

[0086] 2) Configure the Server Node Identifier to identify the gateway node ID and track which gateway node the NVMe command originates from. The Server Node Identifier is an optional setting.

[0087] 3) The Magic Number is used for verification to avoid confusion with other custom Vendor commands with the same opcode. The Magic Number is an optional setting.

[0088] 4) Set the lock mode. The value of the lock mode represents the operation state of the lock. When LockMode is equal to 1, it means to acquire metadata and lock it; when LockMode is equal to 2, it means to release the metadata lock.

[0089] 5) Set the starting logical block address (Starting LBA) to indicate the first logical block, i.e., the metadata starts from this logical block address;

[0090] 6) Set the number of logical blocks, which represents the total number of logical blocks involved in the metadata;

[0091] 7) Set a data pointer to point to the metadata memory of the command sending gateway node. The remote gateway node is responsible for populating the metadata memory. The data pointer field must be populated when Lock Mode is set to 1 (i.e., when acquiring metadata and locking).

[0092] It should be noted that the settings for global namespace identifier, service node identifier, magic number, etc., can be found in the aforementioned implementation example. Additionally, in this example, the Command Dword10 to CommandDword12 fields required for atomic operations are customized using the NVM command set's general format, including the starting logical block address, lock mode, and number of logical blocks. Other fields can be used to define atomic query commands in other examples.

[0093] Therefore, based on the newly defined NVM command set format, a custom NVMe Vendor metadata atomic query command is provided. This allows for global data access to the storage cluster's memory through the aforementioned splitting and forwarding processes when the host needs to perform an atomic metadata query operation. Furthermore, the custom NVMe Vendor metadata atomic query command enables metadata reading and locking to be completed within a single NVMe command, eliminating the need for two consecutive NVMe commands to achieve similar atomic operations, as required by the original NVMe aggregated operations (fused operations).

[0094] In some implementations, gateway node expansion can be performed in a storage gateway cluster based on a custom NVMe Vendor metadata atomic query command. That is, the metadata of the gateway nodes that need to be migrated can be migrated sequentially through the NVMe Vendor metadata atomic query command, which makes the gateway node expansion operation very convenient.

[0095] like Figure 10 As shown, the process of expanding the gateway node is illustrated below:

[0096] 1) Initiate the process of adding a new gateway node X+1;

[0097] 2) Gateway node X+1 reads the cluster configuration and begins initializing the cluster configuration;

[0098] 3) After gateway node X+1 completes configuration initialization, it begins the metadata migration and reconstruction process: it traverses each gateway node in the storage cluster. According to the metadata reconstruction rules, it is determined that gateway node X+1 can share a portion of the metadata from multiple gateway nodes. Then, gateway node X+1 uses the NVMe Vendor metadata atomic query command to migrate the metadata of the gateway nodes that need to be migrated in sequence. During the traversal, it can be determined whether the current gateway node is the last gateway node traversed. If so, the metadata migration is received, thus completing the expansion operation of gateway node X+1. Otherwise, the traversal operation of gateway nodes continues.

[0099] In addition, after the metadata migration is completed, the I / O command process for gateway node X+1 is the same as the service I / O command process for other gateway nodes, and will not be explained further.

[0100] Therefore, online scaling (scale out) can be supported, enabling gateway node expansion without interrupting existing services.

[0101] In some implementations, I / O of metadata for the migrated portion needs to be locked between two gateway nodes using the NVMe Vendor metadata atomic query command. For example, locking can be performed on the storage involved in this metadata using a locking mode. It should be noted that the locking operation can be performed using an existing locking service within the storage cluster; this is not limited to this implementation.

[0102] In some implementations, based on custom NVMe Vendor metadata atomic query commands, data access processing services for the host to the storage cluster's storage can still be provided during gateway node expansion in the storage gateway cluster.

[0103] Specifically, the storage gateway cluster can still provide services for I / O commands from client hosts during the data migration process in the X+1 extension of the gateway node through the aforementioned splitting and forwarding processes.

[0104] The following examples illustrate the scenarios of data hits and misses:

[0105] like Figure 11 As shown, when metadata is hit, the I / O process is as follows:

[0106] a) The client host sends an NVMe Read / Write REQ request command (REQ stands for request, an I / O command request) to a gateway node X+1 that is rebuilding metadata;

[0107] b) Determine whether the LBA region included in the NVMe read / write command of the client host has completed the metadata migration, i.e., whether the metadata is hit;

[0108] c) If a match is found, the gateway node X+1 forwards the NVMe Read / Write REQ request command to the corresponding storage device for read / write operations based on the metadata information.

[0109] d) Gateway node X+1 waits for the NVMe Read / Write RSP response command (RSP stands for response, an I / O command response) sent by the storage device, and after obtaining the read / write results from it, returns the read / write results to the client host.

[0110] like Figure 12 As shown, the I / O process is as follows when metadata is not found:

[0111] a) The client host sends an NVMe Read / Write REQ command request to a gateway node X+1 that is rebuilding metadata;

[0112] b) The metadata of the LBA area contained in the NVMe read / write command queried by gateway node X+1 has not yet been migrated. At this time, gateway node X+1 lacks metadata information such as the mapping relationship between LBA and storage device, which needs to be obtained by other gateway nodes.

[0113] c) Gateway node X+1 populates a custom NVMe Vendor metadata atomic query command, sets Lock Mode=1 to obtain metadata and acquire a lock. Other key information filled in includes Starting LBA, Number of LogicalBlocks, Global Namespace ID, etc. The Data Pointer points to the metadata memory requested by gateway node X+1. Then, the current NVMe Read / Write REQ command is sent to the gateway node containing the metadata (because gateway node X+1 has not yet completed the metadata migration of this gateway node, it needs to complete the service through other gateway nodes), thereby acquiring a distributed lock through the metadata atomic query.

[0114] d) The gateway node containing metadata information returns the metadata information to the requesting gateway node X+1 via the NVMe Read / Write RSP command, and locks the metadata in this LBA range, successfully locking it;

[0115] e) Gateway node X+1 sends NVMe Read / Write REQ command requests to the corresponding storage device for read / write operations based on metadata information;

[0116] f) Gateway node X+1 waits for the NVMe Read / Write RSP response command returned by the storage device, and after obtaining the read / write results from it, returns the read / write results to the client host through the NVMe Read / Write RSP response command;

[0117] g) Gateway node X+1 fills in the custom NVMe Vendor metadata atomic command to release the metadata lock, where Lock Mode=2 indicates the release of the metadata lock, and Starting LBA and Number of Logical Blocks together represent the scope of the released metadata. By releasing the distributed lock of the metadata, the unlock is successful.

[0118] like Figure 13 As shown, when metadata is partially hit and partially missed, the I / O process is as follows:

[0119] a) The client host sends an NVMe Read / Write REQ command request to a gateway node X+1 that is rebuilding metadata;

[0120] b) When the gateway node X+1 queries the NVMe read and write commands, some of the metadata of the LBA area has been migrated, while some has not been fully migrated. At this time, the gateway node X+1 lacks metadata information such as the mapping relationship between some LBAs and storage devices.

[0121] c) For nodes that have been migrated, read and write access can be performed directly by gateway node X+1;

[0122] d) For the incomplete migration, other gateway nodes are required to acquire the metadata. Here, gateway node X+1 populates a custom NVMe Vendor metadata atomic query command, sets Lock Mode=1 to acquire metadata and acquire a lock. Other key information entered includes Starting LBA, Number of Logical Blocks, Global Namespace ID, etc., where the Data Pointer points to the metadata memory requested by gateway node X+1. Then, the current NVMe Read / Write REQ command is sent to the gateway node containing the metadata (because gateway node X+1 has not yet completed the metadata migration for this gateway node, it needs to complete the service through other gateway nodes), thereby acquiring a distributed lock through the metadata atomic query.

[0123] e) The gateway node containing metadata information returns the metadata information to the requesting gateway node X+1 via the NVMe Read / Write RSP command, and locks the metadata in this LBA range, successfully locking it;

[0124] f) Gateway node X+1 sends NVMe Read / Write REQ command requests to the corresponding storage device for read / write operations based on metadata information;

[0125] g) Gateway node X+1 waits for the NVMe Read / Write RSP response command returned by the storage device, and after obtaining the read / write results from it, returns the read / write results to the client host through the NVMe Read / Write RSP response command;

[0126] h) Gateway node X+1 fills in the custom NVMe Vendor metadata atomic command to release the metadata lock, where Lock Mode=2 indicates the release of the metadata lock, and Starting LBA and Number of Logical Blocks together represent the scope of the released metadata. By releasing the distributed lock of the metadata, the unlock is successful.

[0127] It should be noted that the migrated metadata and the incompletely migrated metadata can be read and written by sending an NVMe Read / Write REQ command request to the corresponding storage device, thereby saving the data access process.

[0128] Based on the same inventive concept, this specification also provides a storage cluster data processing apparatus corresponding to the foregoing embodiments, comprising:

[0129] The splitting module is used to split the target data accessing the storage cluster. The target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format. The global namespace identifier is used to locate the valid namespace identifier across nodes.

[0130] The forwarding module is used to enable the target host to access the target storage in the storage cluster through the target node based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data. The target node is the gateway node in the storage gateway cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier.

[0131] It should be noted that by adding a splitting module and a forwarding module to the execution entity, a storage cluster data processing device can be deployed in the corresponding entity. The deployment diagram will not be repeated here.

[0132] Based on the same inventive concept, this specification also provides a storage cluster data processing system corresponding to the foregoing embodiments.

[0133] The storage cluster data processing system includes: a target host (which may be referred to as a client host), a storage gateway cluster, and a storage cluster. The storage gateway cluster includes several gateway nodes, and the storage cluster includes several storage devices (also referred to as storage devices, which are not distinguished in this description). The target host is communicatively connected to at least one gateway node in the storage gateway cluster, and each gateway node in the storage gateway cluster is communicatively connected to the storage cluster. The target host, the storage gateway cluster, or the storage cluster are equipped with the storage cluster data processing method described in any embodiment of this specification.

[0134] In practice, based on the aforementioned custom NVM command set, client hosts can perform global data access processing on the storage devices in the storage cluster by connecting to the gateway node in the existing storage gateway cluster.

[0135] In practice, NVMe over Fabrics networks can be deployed before and after the storage gateway cluster. For example, the NVMe over Fabrics front-end network is located between the host and the storage gateway cluster, and the NVMe over Fabrics back-end network is located between the storage gateway cluster and the storage cluster.

[0136] In one example, the aforementioned storage cluster data processing method can be deployed on a host, a storage gateway cluster, or a smart network interface card, etc. The deployment can be referred to the foregoing description and will not be elaborated further.

[0137] In one example, when the storage cluster data processing method is deployed on the target host, the method is deployed on any of the following resource devices on the target host: virtual machine, operating system, container, or smart network interface card. Deploying on the host allows for the flexible deployment of I / O splitting and I / O forwarding functionality to the client host, enabling data splitting and forwarding directly on this host. This reduces traffic forwarding within the storage gateway cluster, and this deployment method allows for multi-path functionality at the NVMe layer on the client host.

[0138] Based on the same inventive concept, a storage cluster data processing system includes: a request forwarding node, a target host, a storage gateway cluster, and a storage cluster, wherein the target host is communicatively connected to the request forwarding node, the request forwarding node is communicatively connected to at least one gateway node in the storage gateway cluster, each gateway node in the storage gateway cluster is communicatively connected to the storage cluster, and the request forwarding node is deployed with the storage cluster data processing method described in any embodiment of this specification.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the product embodiments described later are relatively simple since they correspond to the methods; relevant parts can be referred to the descriptions in the system embodiments.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method for a storage cluster, characterized in that, include: The target data used to access the storage cluster is split and processed, wherein the target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format, and the global namespace identifier is used to locate the valid namespace identifier across gateway nodes; Based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data, the target host accesses the target storage in the storage cluster through the target node, wherein the target node is the gateway node in the storage gateway cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier. Based on the mapping relationship between the gateway node ID and the cross-node valid namespace identifier located by the global namespace identifier, populate the gateway node ID and the global namespace identifier to split the target data used to access the storage cluster.

2. The storage cluster data processing method according to claim 1, characterized in that, The storage cluster data processing method further includes: setting up a global namespace service, and splitting the target data used to access the storage cluster through the set global namespace service.

3. The storage cluster data processing method according to claim 2, characterized in that, Several local namespace services are set up, and the total capacity of the several local namespace services is equal to the capacity of the global namespace service.

4. The storage cluster data processing method according to claim 1, characterized in that, The target instruction also includes a metadata atomic query command defined based on the NVM command set general format. The metadata atomic query command is configured with a lock mode, a starting logical block address, and a number of logical blocks. The lock mode is used for locking services when performing atomic query operations on metadata. The starting logical block address is used to represent the starting address when performing atomic query operations on metadata. The number of logical blocks is used to represent the number of logical blocks that need to be atomically queried starting from the starting logical block address when performing atomic query operations on metadata.

5. The storage cluster data processing method according to claim 4, characterized in that, The storage cluster data processing method further includes: performing atomic query operations on the storage cluster based on the metadata atomic query command.

6. The storage cluster data processing method according to claim 5, characterized in that, In the atomic query operation of metadata on the storage cluster based on the metadata atomic query command, it is determined whether the metadata has been migrated to the target node; If so, the target host accesses the target storage in the storage cluster through the target node; If not, the lock mode is set to the locking mode, and the metadata atomic query command is sent to other nodes containing the metadata. After the other nodes containing the metadata lock the metadata, the target host accesses the target storage in the storage cluster through the target node to obtain the metadata. After obtaining the metadata, the lock mode is set to the unlocking mode so that the other nodes containing the metadata can unlock the metadata.

7. The storage cluster data processing method according to claim 5, characterized in that, The storage cluster data processing method further includes: when adding a new node, the new node migrates the target metadata that needs to be migrated in sequence based on the metadata atomic query command, wherein the target metadata is metadata located in other nodes.

8. The storage cluster data processing method according to claim 1, characterized in that, The target instruction also includes a service node identifier defined based on the NVM command set general format. The service node identifier is used to identify the node ID that provides the service and / or to locate the node ID that issues the target instruction. And / or, the target instruction also includes a magic number defined based on the NVM command set general format, which is used to verify the global namespace identifier in the target data.

9. A storage cluster data processing method, characterized in that, Deployed in a storage gateway cluster, the storage cluster data processing method includes: The target data is received by the target host through the first gateway node, wherein the first gateway node is a gateway node in the storage gateway cluster that has a communication connection with the target host. The target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format. The global namespace identifier is used to locate the namespace identifier that is valid across nodes. The first gateway node splits the target data for accessing the storage cluster, and based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the split target data, the target host accesses the target storage in the storage cluster through the target node, wherein the target node is the gateway node in the storage gateway cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster corresponding to the cross-node valid namespace identifier located by the global namespace identifier.

10. The storage cluster data processing method according to claim 9, characterized in that, The storage cluster data processing method further includes: Extend a new gateway node into a storage gateway cluster, wherein the new gateway node is configured to perform the following operations: obtain the cluster configuration of the storage gateway cluster and initialize the cluster configuration; after completing the configuration, perform metadata reconstruction processing, wherein the new gateway node shares some metadata from several other gateway nodes.

11. The storage cluster data processing method according to claim 10, characterized in that, The new gateway node sequentially performs migration operations on the metadata that needs to be migrated in the other gateway nodes through a metadata atomic query command. The metadata atomic query command is a metadata atomic query command defined based on the NVM command set general format contained in the target instruction. The metadata atomic query command is configured with a lock mode, a starting logical block address, and a number of logical blocks. The lock mode is used for locking services when performing atomic query operations on metadata. The starting logical block address is used to represent the starting address when performing atomic query operations on metadata. The number of logical blocks is used to represent the number of logical blocks that need to be atomically queried starting from the starting logical block address when performing atomic query operations on metadata.

12. The storage cluster data processing method according to claim 11, characterized in that, In the atomic query operation of metadata on the storage cluster based on the aforementioned atomic metadata query command, the storage cluster data processing method further includes: Determine whether the metadata has been migrated to the new gateway node; If so, the target storage in the storage cluster is accessed through the new gateway node; If not, the lock mode is set to the locking mode, and the metadata atomic query command is sent to other nodes containing the metadata. After the other nodes containing the metadata lock the metadata, the target storage in the storage cluster is accessed through the new gateway node to obtain the metadata. After obtaining the metadata, the lock mode is set to the lock release mode so that the other nodes containing the metadata release the lock on the metadata.

13. The storage cluster data processing method according to claim 9, characterized in that, The storage cluster data processing method further includes: setting up a global namespace service in the storage gateway cluster, and filling in the gateway node ID and global namespace identifier in the global namespace service according to the mapping relationship between the gateway node ID and the cross-node valid namespace identifier located by the global namespace identifier.

14. The storage cluster data processing method according to claim 13, characterized in that, The storage cluster data processing method further includes: setting up several local namespace services in each gateway node of the storage gateway cluster, wherein the total capacity of the several local namespace services is the capacity of the global namespace service.

15. A storage cluster data processing apparatus, characterized in that, include: The splitting module is used to split the target data accessing the storage cluster. The target data contains target instructions, and the target instructions contain a global namespace identifier defined based on the NVM command set general format. The global namespace identifier is used to locate the valid namespace identifier across nodes. The forwarding module is used to enable the target host to access the target storage in the storage cluster through the target node based on the cross-node valid namespace identifier located by the global namespace identifier obtained from the target data. The target node is the gateway node in the storage gateway cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier, and the target storage is the storage in the storage cluster that corresponds to the cross-node valid namespace identifier located by the global namespace identifier.

16. A storage cluster data processing system, characterized in that, include: The target host, the storage gateway cluster, and the storage cluster are provided, wherein the target host is communicatively connected to at least one gateway node in the storage gateway cluster, each gateway node in the storage gateway cluster is communicatively connected to the storage cluster, and the target host, the storage gateway cluster, or the storage cluster are configured with the storage cluster data processing method as described in any one of claims 1-9.

17. The storage cluster data processing system according to claim 16, characterized in that, When the storage cluster data processing method is deployed on the target host, the storage cluster data processing method is deployed on any of the following resource devices on the target host: virtual machine, operating system, container, smart network card.

18. A storage cluster data processing system, characterized in that, include: The system comprises a request forwarding node, a target host, a storage gateway cluster, and a storage cluster, wherein the target host is communicatively connected to the request forwarding node, the request forwarding node is communicatively connected to at least one gateway node in the storage gateway cluster, each gateway node in the storage gateway cluster is communicatively connected to the storage cluster, and the request forwarding node is equipped with a storage cluster data processing method as described in any one of claims 1-9.

Images