Data storage device, storage system and method of providing virtualized storage

Abstract

Data storage devices, storage systems, and methods of providing virtualized storage are disclosed. The methods can include establishing, by a load balancer, a transport connection between a device and a group of networked storage devices; and transferring data between the device and the group of networked storage devices using a transport protocol over the transport connection, wherein the group of networked storage devices includes two or more networked storage devices and the group of networked storage devices is accessed by the device as a logical storage device. A storage device can include a storage medium, a network fabric interface, and a storage controller configured to transfer data between the storage medium and the device via the network fabric interface over the transport connection, wherein the storage controller is configured to share the transport connection with another networked data storage device.

Description

Technical Field

[0001] This disclosure generally relates to storage devices, and more specifically, to systems and methods for fabric-attached storage devices. Background Technology

[0002] Networked storage devices may include a network interface that enables the storage device to communicate directly with a host via a network fabric (such as Ethernet). Networked solid-state drives (SSDs) may use storage protocols (such as Nonvolatile Memory Express Over Fabric (NVMe-oF)) to transfer data over the network fabric.

[0003] The information disclosed in this background section is only intended to enhance the understanding of the background of the invention, and therefore may contain information that does not constitute prior art. Summary of the Invention

[0004] A method for providing virtualized storage may include: establishing a transport connection between a device and a group of networked storage devices via a load balancer; and transmitting data between the device and the group of networked storage devices via the transport connection using a transport protocol, wherein the group of networked storage devices comprises two or more networked storage devices, and the group of networked storage devices is accessed by the device as a logical storage device. The group of networked storage devices may share a network address. At least two networked storage devices in the group of networked storage devices may use different network ports for the network address. The network address may include an Internet Protocol (IP) address. The transport protocol may include Transmission Control Protocol (TCP). The device and the group of networked storage devices may exchange data using the Non-Volatile Memory Fast Over Network (NVMe-oF) protocol. The receive window space of the transport connection may be shared between at least two networked storage devices in the group of networked storage devices. At least two networked storage devices in the group of networked storage devices may, in response to a read command, transmit data to the device within the transport connection using at least a first transport protocol sequence number and a second transport protocol sequence. One networked storage device in a group of networked storage devices can accumulate one or more acknowledgments from the device in response to receiving read data. For write operations, a load balancer can use the transport connection to directly transfer data from the device to one or more networked storage devices in the group of networked storage devices. For write operations, the load balancer can transfer write data from the device to a first networked storage device in the group of networked storage devices, and the first networked storage device in the group of networked storage devices will transfer at least a portion of the write data from the device to a second networked storage device in the group of networked storage devices. The first networked storage device in the group of networked storage devices determines the distribution of write data among the groups of networked storage devices.

[0005] A storage system may include: a first networked storage device; a second networked storage device; and a load balancer connected to the first and second networked storage devices via a network structure. The load balancer is configured to transmit data between the device and the first and second networked storage devices using a shared transport connection between the first and second networked storage devices. The load balancer may be configured to transmit data between the device and the first and second networked storage devices using a shared window space of the transport connection. The first and second networked storage devices may be configured to operate as logical storage devices. The load balancer and the first and second networked storage devices may be configured to share network addresses between the first and second networked storage devices.

[0006] A storage device may include: a storage medium; a network infrastructure interface; and a storage controller configured to: transfer data between the storage medium and the device via a transport connection through the network infrastructure interface, wherein the storage controller is configured to share the transport connection with another networked data storage device. The storage controller may include logic configured to send a message to the other data storage device in response to a read command from the device, and the message includes information enabling the other data storage device to use the transport connection to transfer read data to the device. The storage controller may include logic configured to receive a message from the other data storage device, and the message may include information enabling the data storage device to use the transport connection to transfer read data to the device. The storage controller may include logic configured to determine the distribution of write data between the data storage device and the other data storage device. The storage controller may include logic configured to send read data to a third-party device in response to a read command from the device. Attached Figure Description

[0007] The accompanying drawings are not necessarily drawn to scale and are used throughout the drawings. For illustrative purposes, elements with similar structures or functions are generally designated by the same reference numerals. The drawings are intended only to facilitate the description of the various embodiments described herein. The drawings do not depict every aspect of the teachings disclosed herein and do not limit the scope of the claims. To prevent obscurity, not all components, connections, etc., may be shown, and not all components may have reference numerals. However, the pattern of component configuration can be readily understood from the drawings. The drawings, together with the description, illustrate exemplary embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0008] Figure 1 An embodiment of a storage system according to the present disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and a group of networked storage devices.

[0009] Figure 2 An example embodiment of a storage system according to the present disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and a group of networked storage devices located in chassis, racks, etc.

[0010] Figure 3 Another example embodiment of a storage system according to this disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and groups of networked storage devices, the networked storage devices being distributed in two or more chassis in core data centers, edge data centers, and mobile edge data centers, etc.

[0011] Figure 4 The diagram shows a sequence of example embodiments of a method for allocating transport connections between one or more initiators and two or more networked storage devices according to the present disclosure.

[0012] Figure 5 The diagram illustrates a sequence of an example embodiment of a read operation according to the present disclosure, in which data from a group of networked storage devices can be transmitted to the initiator during the read operation.

[0013] Figure 6 The diagram illustrates a sequence of example embodiments of a write operation according to the present disclosure, in which data from the initiator can be distributed to a group of networked storage devices via a load balancer.

[0014] Figure 7 The diagram shows a sequence of another example embodiment of a write operation according to the present disclosure, in which data from the initiator can be distributed to the group via one of the means in a group of networked storage devices.

[0015] Figure 8 An example embodiment of a load balancer according to this disclosure is shown.

[0016] Figure 9 An example embodiment of a networked storage device according to this disclosure is shown.

[0017] Figure 10 A flowchart illustrating an example embodiment of a method for providing virtualized storage according to the present disclosure is shown. Detailed Implementation

[0018] In some embodiments, fabric-attached storage devices may use one or more transport protocols that may only provide separate connections between the host and multiple storage devices. That is, a 1:1 mapping may exist between the host and each storage device. Therefore, to distribute data among multiple fabricated devices, the host may need to manage multiple transport connections for multiple storage devices. Furthermore, volume management across multiple storage devices may be implemented in the host's software stack. These management tasks, and the resulting increase in the number of input / output (I / O) operations, may lead to increased host resource consumption (such as CPU power, CPU overhead, memory, etc.).

[0019] In some embodiments according to this disclosure, a group of networked storage devices can be virtualized as an aggregated volume, allowing a host to access the group as a single logical storage device. This transparent virtualization can be achieved, for example, by using a load balancer to share network addresses and / or by allocating transport connections across the group of storage devices. In some embodiments, data can be transferred between a host and one or more individual networked storage devices without requiring peer-to-peer data transfers between storage devices within the aggregated group.

[0020] In some embodiments, one or more networked storage devices may be able to transmit data over a network fabric (such as Ethernet, Fibre Channel, unlimited bandwidth, etc.). In some embodiments, the use of a load balancer may enable a group of networked storage devices to share a network address (e.g., an Internet Protocol (IP) address). For example, the same IP address may be used for each storage device in the group, but with a different port number. The shared IP or other network address may be used with one or more transport protocols (such as Transmission Control Protocol (TCP), Remote Direct Memory Access over Converged Ethernet (RDMA) (RoCE), etc.). In some embodiments, the group of hosts and networked storage devices may use the Nonvolatile Memory Express Over Fabric (NVMe-oF) protocol over a network or any other storage protocol that can operate with one or more transport protocols used by the load balancer and storage devices.

[0021] In some embodiments, the use of a load balancer enables a group of networked storage devices to share or allocate transport connections, for example, by allocating transport window space for transport protocols and / or managing sequence numbers, acknowledgments, transport requests, completion responses, etc. For example, distributed transport connections can be used to distribute write data from a host to one or more networked storage devices in the group. Distributed transport connections can also be used to transfer read data from one or more networked storage devices in the group to a host.

[0022] In some embodiments, a transmission connection may be terminated at a networked storage device that can operate as the primary storage device in a group of networked storage devices. The primary storage device may control and / or manage data transfers between itself and the host, and between itself and other storage devices in the group and the host. In some embodiments, a load balancer may transfer write data directly from the host to one or more storage devices in the group. In some embodiments, data transferred directly to one or more storage devices in the group may not be transferred through another storage device in the group. In some embodiments, a load balancer may transfer write data from the host to the primary storage device, which may then forward some or all of the write data to one or more other storage devices in the group. Some embodiments may use a combination of "direct write data transfer" and "forwarded write data transfer".

[0023] Figure 1 An embodiment of a storage system according to the present disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and a group of networked storage devices. Figure 1 The system (or storage system) 100 shown herein can be used, for example, to implement any of the methods, apparatuses, features, techniques, etc., described in this disclosure. System 100 may include a load balancer 104 and two or more networked storage devices (or storage devices) 108, which are configured as a group 106 of devices capable of cooperating as a single logical storage device. Data from one or more hosts 102 may be distributed among the networked storage devices 108, for example, through the operation of one or more shared transport connections 110 as described below. The networked storage devices 108 may include any type of storage device (such as a hard disk drive (HDD), a solid-state drive (SSD), etc., or a combination thereof), each having one or more network fabric interfaces. For example, in some embodiments, one or more of the networked storage devices 108 may be implemented as an Ethernet SSD (ESSD) with an Ethernet interface. In some embodiments, one or more of the networked storage devices 108 may use the NVMe-oF protocol or any other storage protocol that can operate with one or more transport protocols used by the load balancer and the storage devices.

[0024] The networked storage device 108 may be physically located in a single chassis, rack, data center, etc., or distributed among any combination of chassis, racks, data centers, etc. Data centers may include various locations encompassing edge data centers, mobile edge data centers, ad hoc networks, IoT-based networks, etc. In some embodiments, the networked storage device 108 may be distributed across one or more hierarchies of chassis, racks, data centers, etc., or any combination thereof. Furthermore, one or more subsets of the network (e.g., the core data center) may control aspects of one or more sub-networks (e.g., edge networks) within the hierarchy.

[0025] Networked storage device 108 can be connected to load balancer 104 via one or more shared transport connections 110. Load balancer 104 can be connected to one or more hosts 102 via one or more transport connections 112. The one or more shared transport connections 110 and / or 112 can be implemented, for example, using TCP, RoCE, etc. The one or more shared transport connections 110 and / or 112 can be carried on any network structure 114 and / or 116, which can be implemented, for example, using Ethernet, Fibre Channel, unlimited bandwidth, etc., or any combination thereof, and may include any number and / or any configuration of switches, routers, etc.

[0026] In some embodiments, load balancer 104 may establish and / or maintain one or more shared transport connections 110 and / or transport connections 112. In some embodiments, transport connection 112 from host 102 may pass through load balancer 104 without termination and continue as a shared transport connection 110. In some embodiments, transport connection 112 from host 102 may terminate at load balancer 104, which may establish and / or maintain a separate shared transport connection 110 to complete the transport connection between host 102 and networked storage device 108.

[0027] In some embodiments, load balancer 104 may be implemented using one or more commercially available load balancers with minimal or no modifications to the hardware and / or software. In some embodiments, load balancer 104 may be implemented using one or more load balancers, which may include hardware and / or software particularly suitable for implementing one or more of the methods, apparatuses, features, techniques, etc., described in this disclosure.

[0028] In some embodiments, load balancer 104, networked storage device 108, shared transport connection 110, and / or transport connection 112 may operate at a layer (e.g., transport layer) of a system that may at least partially conform to a network model such as the Open Systems Interconnection (OSI) model. In some embodiments, one or more of load balancers 104 may operate at the transport layer of a system that may at least partially conform to a network model such as the TCP / IP model. In some embodiments, a system according to this disclosure may at least partially conform to more than one network model, or may not conform to any network model.

[0029] The following describes some example embodiments of systems, processes, methods, features, techniques, etc., illustrating some possible implementation details according to this disclosure. These examples are provided to illustrate the principles of this disclosure, but the principles are not limited to these implementation details. For example, some embodiments may be described in the context of systems that can use NVMe-oF, TCP, Ethernet, ESSD, etc., but other embodiments according to this disclosure may be implemented using other types of storage, transport, network protocols, storage devices, etc. Furthermore, some example embodiments may be shown using a specific number and / or arrangement of components, device groups, devices within device groups, etc., but other embodiments may use different numbers and / or arrangements.

[0030] Figure 2 An example embodiment of a storage system according to the present disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and a group of networked storage devices located in chassis, racks, etc. Figure 2 The system 120 shown may include one or more load balancers 122 and two or more ESSDs 126, the load balancers 122 and the ESSDs 126 being interconnected via a network structure that may include one or more Ethernet switches 130 and any number of Ethernet connections 132. Figure 2 In the example shown, one or more load balancers 122 may be located separately from Ethernet switches 130 and ESSD 126, which may be housed within chassis 128. However, in some embodiments, one or more of the load balancers 122 may be housed within chassis 128. In other embodiments, the components may be arranged in any other physical configuration.

[0031] One or more initiators 134 can connect to one or more load balancers 122 via any type of network infrastructure 136, which may include, for example, a public and / or private Internet based on Ethernet, Fibre Channel, unlimited bandwidth, or any combination thereof. Initiators 134 (also referred to as hosts) may include any device, system, process, application, etc., that can use system 120 to store data.

[0032] Two or more devices in ESSD 126 can be configured as cooperatively operating groups 124 so that one or more initiators 134 can access group 124 as a single logical storage device. Therefore, data from initiator 134 can be distributed across two or more devices in ESSD 126 in a manner transparent to initiator 134.

[0033] Data from initiator 134 can be distributed within group 124, for example, through the operation of one or more TCP connections that can be shared across group 124 of ESSDs 126. For example, load balancer 122 can establish and / or maintain TCP connections between initiator 134 and group 124 of ESSDs, such that initiator 134 is unaware that the TCP connection is shared by more than one ESSD 126. This can be achieved, for example, by using the same IP address for all ESSDs 126 in group 124 but a different port number for each ESSD 126. Shared or distributed TCP connections can be managed in a consistent manner for input and / or output (I / O) operations, such as by allocating TCP window space, managing TCP sequence numbers, acknowledgments, sending requests, completing responses, etc.

[0034] In some embodiments, NVMe-oF can be used as a storage protocol for data transfer (or data exchange) between one or more of the initiators 134 and group 124 of ESSD 126. That is, the NVMe storage protocol can be used to transfer data, and the NVMe storage protocol can be carried on top of the TCP transport protocol, which can be transmitted using the IP network protocol over an Ethernet fabric (also known as an Ethernet matrix). However, in other embodiments, any other protocol and / or network structure can be used instead of the above-described protocol and / or network structure, or any other protocol and / or network structure besides the above-described protocol and / or network structure can be used.

[0035] Figure 3Another example embodiment of the storage system according to this disclosure is shown, the storage system including one or more load balancers arranged between one or more hosts and a group of networked storage devices distributed in two or more chassis in core data centers, edge data centers, and mobile edge data centers, etc. Figure 3 The system 140 shown may include, with Figure 2 The components shown in System 120 are similar to those shown in the system. However, in Figure 3 In the system 140 shown, device groups 124 of ESSD 126 (which may, for example, share the same IP address) may be distributed across multiple chassis 128 in the core data center 129. Another device group 142 may be distributed across at least one of chassis 128, edge data center 144, and mobile edge data center 146. The network architecture may be hierarchically expanded to include subnet structures 141, 143, and 145. In other embodiments, device groups may be distributed across any combination of any type of chassis, rack, data center, etc., and any number of device groups may be used.

[0036] Figure 4 The diagram shows a sequence of example embodiments of a method for allocating transport connections between one or more initiators and two or more networked storage devices according to the present disclosure. Figure 4 The methods shown can be used, for example, with any system and / or component described in this disclosure (such as, Figures 1 to 3 as well as Figures 8 to 9 (The system and / or components shown are used together.)

[0037] exist Figure 4 In the illustrated embodiment, load balancer 150 may allocate transport connections (e.g., TCP connections) between initiator 152 and networked storage device (e.g., ESSD) 154. The method may begin with load balancer 150 receiving connection request Connect A from initiator-2 in operation 401, connection request Connect B from initiator-1 in operation 402, connection request Connect C from initiator-2 in operation 403, and connection request Connect D from initiator-1 in operation 404. Connection requests may be queued in buffer 156 while load balancer 150 determines how to allocate connection requests to ESSD 154.

[0038] Load balancer 150 may allocate connection requests based on various factors. For example, in some embodiments, load balancer 150 may make connection allocation at least in part based on a connection type load balancing algorithm (such as round-robin scheduling, least connections, random allocation, etc., or any combination thereof). In some embodiments, load balancer 150 may make connection allocation at least in part based on the amount of storage available at each ESSD 154. In some embodiments, ESSD 154 (e.g., ESSD-1, ESSD-2, ESSD-3, and ESSD-4) may form a group of devices that can be accessed as a single logical device. In some embodiments, one or more of ESSD 154 may be associated with one or more additional ESSDs, although not in... Figure 4 As shown, however, one or more additional ESSDs may form a group of devices that can be accessed as a single logical device together with the corresponding one of the ESSDs 154. In some embodiments, the load balancer 150 may allocate connections based on the amount of storage available at the device group associated with one or more of the ESSDs 154. In some embodiments, the load balancer 150 may allocate connections based on any combination of the factors described above and / or other factors.

[0039] like Figure 4 As shown, after determining how to distribute connection requests, load balancer 150 can forward connection requests to ESSD 154. Specifically, connection request Connect B from initiator-1 can be forwarded to ESSD-3 in operation 405, connection request Connect C from initiator-2 can be forwarded to ESSD-1 in operation 406, connection request Connect D from initiator-1 can be forwarded to ESSD-4 in operation 407, and connection request Connect A from initiator-2 can be forwarded to ESSD-2 in operation 408.

[0040] Then, load balancer 150 may wait for an acknowledgment of "connection established" from ESSD 154, and forward the acknowledgments to the corresponding initiator 152, for example, in the order in which they were received from ESSD 154. For example, in response to connection request Connect A, ESSD-2 may send an acknowledgment Established A to load balancer 150 in operation 409, and load balancer 150 may then forward the acknowledgment to initiator-2 in operation 410. This process may continue until the establishment of all four connection requests (Established A to Established D) has been acknowledged to initiator 152.

[0041] Therefore, the transport connection requested by initiator 152 can be established with ESSD 154 as determined by load balancer 150. In some embodiments, the transport connection can be established in a pass-through configuration and terminated at the corresponding ESSD 154.

[0042] exist Figure 4 In the embodiments shown, the operations are example operations and may involve various additional operations not explicitly shown. In some embodiments, some operations in the illustrated operations may be omitted. In some embodiments, operations may be performed by, in addition to Figure 4 Components other than those shown may be used to perform one or more of the operations. Furthermore, in some embodiments, the timing order of the operations may be altered. Additionally, as mentioned above, the figures are not necessarily drawn to scale; therefore, any scale and / or relationship between elements, events, time intervals, etc., may be longer or shorter based on network latency, computational latency, etc.

[0043] Figure 5 The diagram illustrates a sequence of an example embodiment of a read operation according to the present disclosure, in which data from a group of networked storage devices can be transmitted to the initiator during the read operation. Figure 5 The methods shown can be used, for example, with any system and / or component described in this disclosure (such as, Figures 1 to 3 as well as Figures 8 to 9 (The system and / or components shown are used together.)

[0044] exist Figure 5 In the illustrated embodiment, load balancer 160 can allocate a shared transport connection (e.g., a TCP connection) between initiator 162 and device group 166 of networked storage device (e.g., ESSD) 164. Device group 166 can be configured to operate transparently to initiator 162 as a single logical storage device.

[0045] The method may begin when initiator 162 sends a connection request to load balancer 160 (e.g., ...). Figure 5 In operation 500, the load balancer 160 may forward the connection request to ESSD-1. In this example embodiment, ESSD-1 may operate as the master device of a device group that may include ESSD-1, ESSD-2, ESSD-3, and ESSD-4. In operation 502, ESSD-1 may send an acknowledgment to the load balancer 160 indicating that a TCP connection has been established (e.g., ...). Figure 5(The "established" part is mentioned in the original text). Then, in operation 503, load balancer 160 can forward the acknowledgment to initiator 162. In this example embodiment, a TCP connection can be established through load balancer 160 in a pass-through configuration, which can terminate at ESSD-1 but has separate port spaces. That is, all four ESSDs 164 in the group can use the same IP address but use different port numbers to avoid conflicts between ESSDs when receiving and / or sending over the TCP connection.

[0046] When initiator 162 sends a read command in operation 504, the read operation can begin. In operation 505, load balancer 160 can forward the read command to ESSD-1. The read command may include one or more parameters (such as Logical Block Address (LBA), Number of Blocks to Read (LEN), Receive Window Size (RCV WIN), etc.). In this example, the read command from initiator 162 may specify the reading of 32 blocks (LEN = 32), where each block may be, for example, 512 bytes starting from LBA 0.

[0047] In response to a read command, ESSD-1 can determine where the requested read data is stored in group 166 of ESSD 164 by consulting, for example, a mapping table, array, and / or other data structures. The mapping data structure may be located at and / or maintained by the master ESSD-1, for example. In some embodiments, a copy of the mapping data structure may also be located at and / or maintained by any other ESSD in load balancer 160 and / or device group 166.

[0048] In this example, the data requested by the read command can be distributed across all four ESSDs in device group 166. To initiate the process of transferring data from ESSD 164 to initiator 162, ESSD-1 can send one or more read messages to each of the ESSDs 164 that may have data in response to the read command. The read message may include one or more parameters (such as the Read Logical Block Address (READ LBA), the number of blocks to read (LEN), the TCP sequence number (Seq), the destination IP address (DST IP), etc.). Figure 5 Examples of read message parameters that can be sent to ESSD-2, ESSD-3, and ESSD-4 in operations 506, 507, and 508, respectively, are shown.

[0049] In some embodiments, the READ LBA included in a read message to an ESSD may be the ESSD's local LBA. That is, the mapping data structure allows ESSD-1 (the primary ESSD) to tell each other ESSD in group 166 where the requested data is located in its own storage medium. In some embodiments, for example, where a copy of the mapping data structure can be maintained at each ESSD 164, the ESSD may be able to determine its local LBA based on the global LBA of group 166.

[0050] Using parameters sent along with the read message (such as TCP sequence number and destination IP address), each ESSD in group 166 can read the requested data from its storage medium and directly transmit the read data to initiator 162 via a distributed TCP connection through load balancer 160. For example, this is illustrated in operation 509, where ESSD-1 can send approximately 4096 bytes of data (eight blocks of approximately 512 bytes each) to load balancer 160 using TCP sequence number 0, and then, in operation 510, load balancer 160 can forward the data to initiator 162. In some embodiments, a storage protocol (such as NVMe-oF combined with TCP, for example) can be used to transmit the read data.

[0051] In this example, ESSD-3, ESSD-2, and ESSD-4 can each send their eight data blocks using TCP sequence numbers 8192, 4096, and 12288 respectively in operations 511, 515, and 519. Load balancer 160 can then forward the data blocks to initiator 162 in operations 512, 516, and 520 respectively. Therefore, the TCP receive window space can be distributed among the different ESSDs 164. In one example, the TCP receive window space can be shared among the different ESSDs 164. The initial read command from initiator 162 can specify the read of approximately 32 blocks (LEN = 32), where each block can be approximately 512 bytes, totaling approximately 16KB. Therefore, four data transmissions resulting from approximately 4096 bytes (Len = 4096) from each ESSD in group 166 can send a total of approximately 16KB.

[0052] When initiator 162 receives read data from ESSD, it can send an acknowledgment (ACK) back to load balancer 160, as shown in operations 513, 517, and 521. This acknowledgment can then be forwarded in operations 514, 518, and 522, respectively. In one example, in response to initiator 162 receiving read data, ESSD-1 accumulates one or more acknowledgments from initiator 162. After ESSD-1 receives the final ACK in operation 522, in operation 523, ESSD-1 can send a read complete response. In operation 524, the read complete response can be forwarded to initiator 162, thus completing the read operation.

[0053] Based on the order in which data is transmitted and read according to ESSD 164, the initiator 162 can use TCP sequence numbers to rearrange the received data to restore the data to its original order.

[0054] In some embodiments, an I / O command context may be created and / or maintained, for example, at load balancer 160 and / or ESSD-1. For instance, a command context may be created when a read command is received at the start of a read operation and then discarded after the read command is completed. The command context may contain information that keeps track of "which ESSDs 164 contain data in response to the read command, which ESSDs 164 have sent their read data, which acknowledgments have been received from initiator 162, etc."

[0055] In some embodiments, at least a portion of the read data may be sent to a location other than the initiator 162. For example, in some embodiments, a read command from the initiator 162 may include a destination IP address that can be used by a third party different from the initiator 162. In such an embodiment, the third-party destination IP address may be passed to ESSD 164 in the read message, and ESSD 164 may then send the read data directly to the third-party destination IP address via load balancer 160. Such embodiments may be useful, for example, in video streaming and / or gaming applications, where the initiator 162 may operate as a metadata server to manage the flow of streaming data to third-party users.

[0056] exist Figure 5 In the embodiments shown, the operations are example operations and may involve various additional operations not explicitly shown. In some embodiments, some operations in the illustrated operations may be omitted. In some embodiments, operations may be performed by, in addition to Figure 5Components other than those shown in the diagram may be used to perform one or more of the operations. For example, in some embodiments, one or more read messages may be sent from load balancer 160 instead of the primary ESSD, or one or more read messages may be sent from load balancer 160 in addition to the primary ESSD. As another example, in some embodiments, at least some of the acknowledgments may be accumulated by load balancer 160 instead of the primary ESSD, or at least some of the acknowledgments may be accumulated by load balancer 160 in addition to the primary ESSD. Additionally, in some embodiments, the timing order of the operations may be changed. Furthermore, as mentioned above, the figures are not necessarily drawn to scale, and therefore any scale and / or relationship between elements, events, time intervals, etc., may be longer or shorter based on network latency, computation latency, etc.

[0057] Figure 6 The diagram illustrates a sequence of example embodiments of write operations according to this disclosure, wherein data from the initiator can be distributed by a load balancer to a group of networked storage devices. Figure 6 The methods shown can be used, for example, with any system and / or component described in this disclosure (such as, Figures 1 to 3 as well as Figures 8 to 9 (The system and / or components shown are used together.)

[0058] exist Figure 6 In the embodiments shown, it can be used with Figure 5 The embodiment shown illustrates a similar approach in which a load balancer 170 establishes a distributed transport connection between the initiator 172 and the device group 176 of the networked storage device (e.g., ESSD) 174. This is also similar to... Figure 5 The embodiment shown, Figure 6 The device group 176 shown can be configured to operate transparently to the initiator 172 as a single logical storage device, and ESSD-1 can operate as the master device of the device group 176, which may include ESSD-1, ESSD-2, ESSD-3 and ESSD-4.

[0059] The method may begin when initiator 172 sends a connection request to load balancer 170 (e.g., ...). Figure 6 In operation 600, the load balancer 170 may send a connection request and hash to ESSD-1. In operation 602, ESSD-1 may send an acknowledgment to the load balancer 170 indicating that a TCP connection has been established (e.g., ...). Figure 6(The "established" part is mentioned in the original text). Then, in operation 603, load balancer 170 can forward the acknowledgment to initiator 172. When initiator 172 sends a write command in operation 604, the write operation can begin, and in operation 605, load balancer 170 can forward the write command to ESSD-1. The write command can include one or more parameters (such as the logical block address to write (WRITE LBA), the number of blocks to read (LEN), the receive window size (RCV WIN), etc.). In this example, the write command from initiator 172 can specify the writing of approximately 32 blocks (LEN=32), where each block can be, for example, approximately 512 bytes starting from LBA 0.

[0060] In some embodiments, ESSD-1 may send one or more data arrival notifications to ESSD-2, ESSD-3 and ESSD-4 respectively in operations 606, 607 and 608.

[0061] In operation 609, ESSD-1 may send a Ready to Transmit (R2T) notification to load balancer 170, and in operation 610, the R2T notification may be forwarded to initiator 172. The R2T notification may include one or more parameters (such as an indicator for RAID-0 (Redundant Array of Independent Disks-0) (1 stripe) configured for ESSD group 176, receive window size (RCV WIN), etc.).

[0062] In operation 611, initiator 172 may send write data 178, which may include a header and a data portion, to load balancer 170, which may store the write data in buffer 180. The write data may be accompanied by one or more parameters (such as write data length (WR DATA LEN) and / or other parameters). In some embodiments, a storage protocol (such as NVMe-oF combined with TCP, for example) may be used to transmit the write data.

[0063] In operation 612, load balancer 170 may determine the demarcation and / or distribution of write data among ESSD 174. The demarcation and / or distribution may be based on various factors. For example, in some embodiments, load balancer 170 may make the demarcation and / or distribution at least in part based on the amount of storage available at each ESSD 174. In some embodiments, load balancer 170 may make the demarcation and / or distribution at least in part based on various load balancing algorithms (such as round-robin scheduling, least connections, random distribution, etc., or any combination thereof). In some embodiments, load balancer 170 may make the demarcation and / or distribution based on any combination of the above factors and / or other factors.

[0064] After determining the partitioning and / or distribution of the write data, the load balancer 170 can directly transmit portions 182, 184, 186, and 188 of the write data to ESSD-1, ESSD-2, ESSD-3, and ESSD-4 via the allocated transport connections in operations 613, 614, 615, and 616, respectively. Each of the write data portions 182, 184, 186, and 188 may include a corresponding header and data portion. In this example, the write data is distributed among all four ESSDs 174 in group 176; however, in other cases, the write data can be distributed among any number of ESSDs 174 in any manner.

[0065] In some embodiments, ESSD-1 may accumulate data reception acknowledgments that can be received in operations 617, 618, and 619 in buffer 189. Then, in operation 620, ESSD-1 may send a write completion response to load balancer 170, and then, in operation 621, the write completion response may be forwarded to initiator 172, thereby completing the data write operation.

[0066] exist Figure 6 In the embodiments shown, the operations are example operations and may involve various additional operations not explicitly shown. In some embodiments, some operations in the illustrated operations may be omitted. In some embodiments, operations may be performed by, in addition to Figure 6 Components other than those shown may be used to perform one or more of the operations. For example, in some embodiments, data arrival notification may be omitted. As another example, in some embodiments, data reception acknowledgments may be accumulated at load balancer 170 instead of at ESSD-1, or data reception acknowledgments may be accumulated at load balancer 170 in addition to ESSD-1. Furthermore, in some embodiments, the timing order of operations may be changed. Additionally, as mentioned above, the figures are not necessarily drawn to scale, and therefore any scale and / or relationship between elements, events, time intervals, etc., may be longer or shorter based on network latency, computation latency, etc.

[0067] Figure 7 The diagram shows a sequence of another example embodiment of a write operation according to the present disclosure, in which data from the initiator can be distributed to the group by one of the devices in a group of networked storage devices. Figure 7 The methods shown can be used, for example, with any system and / or component described in this disclosure (e.g., Figures 1 to 3 as well as Figures 8 to 9 (The system and / or components shown are used together.)

[0068] exist Figure 7 In the embodiments shown, it is possible to use with respect to... Figure 5 and Figure 6The embodiment shown illustrates a similar approach, where a distributed transport connection is established between the initiator 192 and the device group 196 of the networked storage device (e.g., ESSD) 194 via a load balancer 190. This is also similar to... Figure 5 and Figure 6 The embodiment shown, Figure 7 The device group 196 shown can be configured to operate transparently to the initiator 192 as a single logical storage device, and ESSD-1 can operate as the master device of the device group 196, which may include ESSD-1, ESSD-2, ESSD-3 and ESSD-4.

[0069] Operations 700 to 703 can be used with Figure 6 The operations shown in 600 to 603 are similar. Figure 7 In the illustrated embodiment, a write operation can begin when initiator 192 sends a write command in operation 704, and in operation 705, load balancer 190 can forward the write command to ESSD-1. The write command may include one or more parameters (such as the logical block address to write (WRITELBA), the number of blocks to read (LEN), the receive window size (RCV WIN), etc.). In this example, the write command from initiator 192 may specify the writing of approximately 32 blocks (LEN = 32), where each block may be, for example, approximately 512 bytes starting from LBA 0. In some embodiments, a storage protocol (such as NVMe-oF combined with TCP, for example) may be used to transmit the write data.

[0070] In operation 706, ESSD-1 determines how much (if any) buffer space it has available to receive write data from initiator 192. In operation 707, ESSD-1 sends a Ready to Transmit (R2T) notification to load balancer 190, and in operation 708, the R2T notification is forwarded to initiator 192. The R2T notification may include one or more parameters (such as an indicator that ESSD group 196 can be configured for RAID-0 (1 stripe), receive window size (RCV WIN), etc.). In some embodiments, one or more parameters may include information about how much buffer space ESSD-1 has available.

[0071] In operation 709, initiator 192 may send write data 197, which may include a header and a data portion, to load balancer 190. In operation 710, load balancer 190 may forward the write data to ESSD-1. The write data may be accompanied by one or more parameters (such as write data length (WR DATA LEN) and / or other parameters). The primary storage device ESSD-1 may then store the write data in buffer 189 and determine the partitioning and / or distribution of the write data among ESSDs 174.

[0072] ESSD-1 can be based on and Figure 6 In the illustrated embodiment, partitioning and / or distribution are determined by various factors similar to those used by load balancer 170. After the partitioning and / or distribution of the write data is determined, master drive ESSD-1 may reserve a portion of the write data for storage using its own storage medium, and / or transfer multiple portions of the write data to ESSD-2, ESSD-3, and / or ESSD-4 respectively in operations 711, 712, and 713. In some embodiments, ESSD-1 may accumulate or wait in buffer 199 for write completion acknowledgments that may be received in operations 714, 715, and 716. Then, in operation 717, ESSD-1 may send a write completion response to load balancer 190, which may then be forwarded to initiator 192 in operation 718, thereby completing the data write operation.

[0073] exist Figure 7 In the embodiments shown, the operations are example operations and may involve various additional operations not explicitly shown. In some embodiments, some operations in the illustrated operations may be omitted. In some embodiments, operations may be performed by, in addition to Figure 7 Components other than those shown may be used to perform one or more of the operations. For example, in some embodiments, write completion acknowledgments may be accumulated at load balancer 190 instead of at ESSD-1, or write completion acknowledgments may be accumulated at load balancer 190 in addition to ESSD-1. Furthermore, in some embodiments, the timing order of the operations may be changed. Additionally, as mentioned above, the figures are not necessarily drawn to scale, and therefore any scale and / or relationship between elements, events, time intervals, etc., may be longer or shorter based on network latency, computation latency, etc.

[0074] In some embodiments, the transmission of information (such as notifications, write data, read data, read messages, acknowledgments, completions, etc.) between drives in the device group can be performed via any suitable communication channel. For example, in some embodiments, one or more separate protocols can be used to implement one or more Virtual Local Area Networks (VLANs) for the storage devices in the group. Such one or more protocols can be carried by a network structure interconnected with the storage devices in the group and carrying shared transport connections and / or by any other communication channel (such as a separate physical network). In some embodiments, these one or more additional protocols can be used to allocate reserved bandwidth for the device group.

[0075] In some embodiments, a low-latency communication protocol, such as embedded remote procedure call (eRPC), can be used to transmit command and / or response information between storage devices in a group. This can be used for example... Figure 5 The read message shown, and / or Figure 6 and Figure 7 The data arrival notification and / or completion acknowledgment shown are illustrated. Example implementations of reading messages may include, for example, any number of the following parameters: VLAN identifier, source IP address, destination IP address, TCP sequence number, transmission length, the block read (which may include, for example, namespace, block address, length), and other TCP and / or IP flags requested by the master device.

[0076] In some embodiments, read and / or write data transfers between storage devices in a group can be performed, for example, via VLANs that can use protocols such as RDMA (e.g., RoCE). In some embodiments, a Quality of Service (QoS) environment can be established to avoid potential performance issues.

[0077] Figure 8 An example embodiment of a load balancer according to this disclosure is shown. Figure 8 The load balancer 200 shown can be used, for example, to implement any of the methods, devices, features, etc., described in this disclosure. The load balancer 200 may include one or more network interface controllers (NICs) 202, which can be used to establish network connections, for example, between any number of initiators, switches, networked storage devices, etc. A switch fabric 204 can establish and maintain network connections between network devices connected to the NIC 202. A connection management module 206 can control the establishment, maintenance, termination, mapping, etc., of connections between network devices at the network layer, transport layer, etc. Storage management logic 208 can implement any storage-related functions and / or features according to this disclosure. For example, if the load balancer 200 is used to implement… Figure 6The method shown allows storage management logic 208 to be used to determine the partitioning and / or distribution of write data among multiple ESSDs in a device group.

[0078] Figure 8 Any component and / or function of the embodiments illustrated may be implemented using hardware, firmware, software, or any combination thereof. Furthermore, any component among the various components may be implemented as a separate component, integrated together, or arranged in any other suitable configuration. For example, in some embodiments, the switching fabric 204 may be implemented using a hardware switch, while in other embodiments, some or all of the switching fabric may be implemented in software. In some embodiments, the load balancer 200 may be implemented by adding storage management logic 208 to an existing network load balancer.

[0079] Figure 9 An example embodiment of a networked storage device according to this disclosure is shown. Figure 9 The storage device 210 shown can be used, for example, to implement any of the methods, apparatuses, features, etc., described in this disclosure. The storage device 210 may include one or more network interfaces (also referred to as "network fabric interfaces") 212, a storage controller 214, and a storage medium 218. The storage medium 218 may be implemented using magnetic, solid-state, optical, or any other type of data storage technology. The one or more network interfaces 212 enable the storage device to connect to a network fabric based on, for example, Ethernet, Fibre Channel, unlimited bandwidth, etc. The storage controller 214 controls the overall operation of the storage device 210. The storage controller 214 may include logic 216 that can be used to implement any of the functions and / or features according to this disclosure. For example, if the storage device 210 is used to implement… Figure 5 For any storage device in the illustrated device group, logic 216 may include functionality for sending read messages to other devices to instruct them to send read data to the initiator, or functionality for other ESSDs to receive read messages and respond accordingly. As another example, if storage device 210 is used to implement... Figure 6 For any storage device in the device group shown, logic 216 may include functionality for determining a master ESSD for partitioning and / or distributing write data among the ESSDs in the device group. As another example, logic 216 may include functionality for sending read data to a third-party device in response to a read command from an initiator.

[0080] Network interface 212 and storage controller 214 may be implemented as separate components or integrated into one or more components using hardware, software, or any combination thereof. For example, storage controller 214 may be implemented using any suitable data processing device including microprocessors, microcontrollers, etc., and may be implemented as a system-on-a-chip (SOC), field-programmable gate array (FPGA), a combination of standard and / or custom integrated circuits (ICs), application-specific integrated circuits (ASICs), etc., or any combination thereof.

[0081] Figure 10 A flowchart illustrating an example embodiment of a method for providing virtualized storage according to the present disclosure is shown. In operation 222, Figure 10 The method 220 shown can be started. In operation 224, a transport connection can be established between the device and the group of networked storage devices via a load balancer. In operation 226, data can be transferred between the device and the group of networked storage devices via the transport connection using a transport protocol. In some embodiments, the group of networked storage devices may include two or more networked storage devices and can be accessed by the device as logical storage devices. In operation 228, the method can end.

[0082] exist Figure 10 In the embodiments shown, the operations are example operations and may involve various additional operations not explicitly shown. In some embodiments, some operations in the illustrated operations may be omitted. In some embodiments, operations may be performed by, in addition to Figure 10 Components other than those described herein may be used to perform one or more of the operations. Furthermore, in some embodiments, the timing order of the operations may be altered.

[0083] In some embodiments, depending on implementation details, one or more systems, methods, and / or apparatuses according to this disclosure can overcome one or more potential problems with networked storage devices. For example, some protocols may provide only one transport connection per device / drive pair. Therefore, to extend storage across multiple networked devices, a host central processing unit (CPU) can handle processing related to managing volumes across multiple devices. This can also increase the number of I / O operations handled by the host CPU. However, in some embodiments, depending on implementation details, one or more systems, methods, and / or apparatuses according to this disclosure can enable multiple networked storage devices to be accessed by the host CPU as a single logical device. Furthermore, in some embodiments, depending on implementation details, one or more systems, methods, and / or apparatuses according to this disclosure can enable networked storage devices to be virtualized as aggregated volumes without any changes to transport protocols such as TCP and / or RDMA.

[0084] The embodiments disclosed herein may be described in the context of various implementation details, but the principles of this disclosure are not limited to these or any other specific details. Some functions have been described as being implemented by certain components, but in other embodiments, functions may be distributed among different systems and components in different locations and have various user interfaces. Specific embodiments have been described as having specific processes, steps, combinations thereof, etc., but these terms may also include embodiments in which specific processes, steps, combinations thereof, etc. may be implemented by multiple processes, steps, combinations thereof, etc., or multiple processes, steps, combinations thereof, etc. may be integrated into a single process, step, combination thereof, etc. References to components or elements may refer only to a portion of a component or element. Unless otherwise clear from the context, the use of terms such as “first” and “second” in this disclosure and claims is for the purpose of distinguishing what they modify only and may not indicate any spatial or temporal order. References to a first item may not imply the presence of a second item. In some embodiments, a networked storage device may only make a partition or other portion of the storage medium of the device available for aggregation in a group of devices. Thus, in some embodiments, a storage device may also refer to a partition or other portion within the device. Furthermore, the various details and embodiments described above can be combined to produce additional embodiments based on the inventive principles disclosed in this patent.

[0085] Since modifications can be made to the inventive principles disclosed in this patent in terms of arrangement and details without departing from the inventive concept, such changes and modifications are considered to fall within the scope of the claims.

Claims

1. A method for providing virtualized storage, the method comprising: A transport connection is established between a host and a group of networked storage devices via a load balancer, wherein the load balancer is positioned between the host and the group of networked storage devices, the group of networked storage devices is connected to the load balancer via a shared first transport connection, and the load balancer is connected to the host via a second transport connection; and Data is transmitted between the host and the group of networked storage devices via the transport connection using a transport protocol. In this context, a group of networked storage devices comprises two or more networked storage devices, and the group of networked storage devices is virtualized as an aggregated volume, enabling the host to access the group of networked storage devices as a single logical storage device. In this group of networked storage devices, two or more networked storage devices share a network address and use different network ports for that network address. The load balancer, the two or more networked storage devices, the first transmission connection, and the second transmission connection all operate at the transport layer. The method further includes: Based on the write command, using the network address and the first network port number, the first part of the data is stored in the first of the two or more networked storage devices; Based on the write command, using the network address and the second network port number, the second portion of the data is stored in the second of the two or more networked storage devices.

2. The method according to claim 1, wherein, The network address includes Internet Protocol (IP) addresses.

3. The method according to claim 1, wherein, Transmission protocols include transmission control protocols.

4. The method according to claim 1, wherein, The host and networked storage devices use a network-based non-volatile memory fast protocol to exchange data.

5. The method according to claim 1, wherein, The receiving window space of the transmission connection is shared among at least two networked storage devices in the group of networked storage devices.

6. The method according to claim 1, wherein, At least two networked storage devices in a group of networked storage devices, in response to a read command, transmit data to a host within the transport connection using at least a first transport protocol sequence number and a second transport protocol sequence number.

7. The method according to any one of claims 1 to 6, wherein, In response to the host receiving read data, one of the networked storage devices in a group of networked storage devices accumulates one or more acknowledgments from the host.

8. The method according to any one of claims 1 to 6, wherein, For write operations, the load balancer uses the transport connection to transfer data directly from the host to one or more networked storage devices in a group of networked storage devices.

9. The method according to any one of claims 1 to 6, wherein, For write operations: The load balancer transfers write data from the host to the first networked storage device in the group of networked storage devices; and The first networked storage device in the group of networked storage devices transfers at least a portion of the written data from the first networked storage device to the second networked storage device in the group of networked storage devices.

10. The method according to claim 9, wherein, The first networked storage device in a group of networked storage devices determines the distribution of written data among the multiple networked storage devices in the group of networked storage devices.

11. A storage system, comprising: First networked storage device; Second networked storage device; as well as A load balancer is positioned between the host and a first networked storage device and a second networked storage device. The first and second networked storage devices are connected to the load balancer via a shared first transmission connection, and the load balancer is connected to the host via a second transmission connection. The load balancer is configured to: establish a shared first transmission connection and a second transmission connection; and use the first and second transmission connections to transfer data between the host and the first and second networked storage devices. In this configuration, the first networked storage device and the second networked storage device are virtualized into a converged volume, enabling the host to access the first networked storage device and the second networked storage device as a single logical storage device. The first networked storage device and the second networked storage device share a network address, but use different network ports for that network address. The load balancer, the first networked storage device, the second networked storage device, the first transmission connection, and the second transmission connection all operate at the transport layer. The first networked storage device is configured to store a first portion of data based on a write command, using a network address and a first network port number. The second networked storage device is configured to store a second portion of the data based on a write command, using a network address and a second network port number.

12. The storage system according to claim 11, wherein, The load balancer is configured to transfer data between the host and the first and second networked storage devices using a shared window space of the first transmission connection.

13. A data storage device, comprising: Storage medium; Network structure interface; as well as The storage controller is configured to transfer data between the storage medium and the host via a network interface through a first transmission connection. The storage controller is configured to share a first transmission connection with another networked data storage device. The load balancer is positioned between the host and the data storage device and the other data storage device. The data storage device and the other data storage device are connected to the load balancer via a shared first transmission connection, and the load balancer is connected to the host via a second transmission connection. The data storage device and the other data storage device are virtualized into a aggregated volume, allowing the host to access the data storage device and the other data storage device as a single logical storage device. The data storage device and the other data storage device share a network address, but use different network ports for that network address. The load balancer, the data storage device, the other data storage device, the first transmission connection, and the second transmission connection all operate at the transport layer. The storage controller includes logic configured to determine the distribution of written data between the data storage device and the other data storage device. The first part of the data is stored in the data storage device based on the network address and the first network port number; The second part of the data is stored in the other data storage device based on the network address and the second network port number.

14. The data storage device according to claim 13, wherein, The storage controller includes logic configured to send a message to the other data storage device in response to a read command from the host, and The message includes information that enables the other data storage device to transmit read data to the host using the first transmission connection.

15. The data storage device according to claim 13, wherein, The storage controller includes logic configured to receive messages from the other data storage device, and The message includes information enabling the data storage device to transmit read data to the host using the first transmission connection.

16. The data storage device according to any one of claims 13 to 15, wherein, The storage controller includes logic configured to determine the distribution of written data between the data storage device and the other data storage device.

17. The data storage device according to any one of claims 13 to 15, wherein, The storage controller includes logic configured to send read data to a third-party device in response to a read command from the host.

Images