Method and system for a peer-to-peer data channel for storage devices

Abstract

A method may include transferring data between a host and a first storage device via a first storage interface, transferring data between the host and the second storage device via a second storage interface, and transferring data between the first storage device and the second storage device via a peer-to-peer channel. A storage system may include a host interface, a first storage device having a first storage interface coupled to the host interface, a second storage device having a second storage interface coupled to the host interface, and a peer-to-peer bus coupled between the first and second storage devices. The storage device may include a storage medium, a storage device controller coupled to the storage medium, a storage interface coupled to the storage device controller, and a peer-to-peer interface coupled to the storage device controller.

Description

Technical Field

[0001] The present disclosure relates generally to data storage, and more particularly, to systems and methods for storing data in storage devices, data centers, edge centers, and the like. Background Art

[0002] The storage system may include one or more storage devices, such as hard disk drives (HHDs), solid-state drives (SSDs), etc., located in one or more chassis, racks, etc. Data may be transmitted between the storage device and one or more local or remote hosts through a storage interface on each storage device and / or host. The storage device may be connected to the storage system by, for example, plugging a connector on the device into a corresponding connector on a midplane, a switchboard, etc. The storage system may also include a management component, such as a baseboard management controller (BMC), which may perform configuration, monitoring, and / or other management functions of the storage system. Summary of the invention

[0003] A method may include transmitting data between a host and a first storage device through a first storage interface, transmitting data between the host and the second storage device through a second storage interface, and transmitting data between the first storage device and the second storage device through a peer-to-peer channel. The method may also include discovering a topology of the peer-to-peer channel. Discovering the topology may be performed at least in part by one or more of the host, the management controller, or at least one of the storage devices. The method may also include configuring quality of service (QoS) for the first and second storage devices. The storage device may be configured with QoS by one or more of the host, the management controller, or at least one of the storage devices. The QoS may include one or more of priority, resource, or performance level. The method may also include transmitting data between the host and the third storage device through a third storage interface, and configuring the first, second, and third storage devices into one or more peer groups. Peer groups may be created dynamically. Peer groups may be configured for one or more of one-to-one, one-to-many, or many-to-many communications. At least one of the storage devices may autonomously perform at least one of reliability, availability, or security functions from the host.

[0004] The storage system may include a host interface, a first storage device having a first storage interface coupled to the host interface, a second storage device having a second storage interface coupled to the host interface, and a peer bus coupled between the first and second storage devices. The peer bus may include one or more of a wired bus, a wireless bus, or an optical bus. The first storage interface may be coupled to the host interface via a first storage connector, the second storage interface may be coupled to the host interface via a second storage connector, and the peer bus is coupled between the first storage connector and the second storage connector. The peer bus may include one or more passive connections. The peer bus may include one or more active connections. The peer bus may include a switching structure. The peer bus may be at least partially located on one of a midplane, a motherboard, or a switch board. The first storage device may be coupled to the peer bus via a first peer connector, and the second storage device may be coupled to the peer bus via a second peer connector. The peer bus may include one or more contacts arranged to couple the first peer connector to the peer bus. The one or more contacts may include one or more sliding contacts. The peer bus may include one or more of a board, a circuit board, a ribbon, or a cable. The system may be contained in an enclosure comprising one or more of a cabinet, a rack, a room, or a building.

[0005] The storage device may include a storage medium, a storage device controller coupled to the storage medium, a storage interface coupled to the storage device controller, and a peer interface coupled to the storage device controller. The peer interface may include one or more of a wired interface, a wireless interface, or an optical interface. The storage interface may include a two-port interface. The storage interface may be capable of loopback operations. A storage connector may be coupled to the storage interface. The storage connector may be coupled to the peer interface. The storage device may also include an additional connector coupled to the peer interface. The additional connector may include one or more sliding contacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The drawings are not necessarily drawn to scale, and for illustrative purposes, elements of similar structure or function are generally represented by similar reference numerals throughout the drawings. The drawings are intended only to facilitate the description of the various embodiments described herein. The drawings do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. In order to prevent the drawings from becoming obscure, not all components, connections, etc. may be shown, and not all components may have reference numerals. However, the mode of component configuration may be apparent from the drawings. The drawings, together with the specification, illustrate example embodiments of the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0007] Figure 1 An embodiment of a storage system according to the present disclosure is shown.

[0008] Figure 2 Another embodiment of a storage system according to the present disclosure is shown.

[0009] Figure 3 An embodiment of a storage system according to the present disclosure is shown, which may include two or more storage devices that may have peer-to-peer data transfer capabilities.

[0010] Figure 4 An embodiment of a storage device according to the present disclosure that may have peer-to-peer data transfer capabilities is shown.

[0011] Figure 5 An example embodiment of a storage system having a storage device with peer-to-peer data transfer capabilities implemented using wireless technology according to the present disclosure is shown.

[0012] Figure 6 Another example embodiment of a storage system having a storage device with peer-to-peer data transfer capabilities implemented using wireless technology according to the present disclosure is shown.

[0013] Figure 7 An example embodiment of a storage system having a storage device with peer-to-peer data transmission capabilities implemented using optical communication technology according to the present disclosure is shown.

[0014] Figure 8 An example embodiment of a mechanical configuration of a storage system having a storage device with peer-to-peer data transfer capabilities implemented using wireless technology according to the present disclosure is shown.

[0015] Fig. 9 An example embodiment of an electrical configuration for a wired peer-to-peer bus for a storage system according to the present disclosure is shown.

[0016] Fig.10 An example embodiment of a storage system having storage devices with peer-to-peer data transfer capabilities implemented using a passive wired bus connected through existing connectors according to the present disclosure is shown.

[0017] Fig.11 An example embodiment of a storage system having storage devices with peer-to-peer data transfer capabilities implemented using an active wired bus connected through existing connectors according to the present disclosure is shown.

[0018] Fig.12 An example embodiment of a message format according to the present disclosure is shown.

[0019] Fig.13A data flow diagram illustrating an embodiment of a method for operating a storage system in accordance with the present disclosure in which storage devices may exchange data through one or more peer-to-peer channels.

[0020] Fig.14 An embodiment of a computing system according to the present disclosure is shown. DETAILED DESCRIPTION

[0021] In some embodiments according to the present disclosure, two or more storage devices can exchange data through one or more peer-to-peer channels without directly involving local and / or remote hosts. Such peer-to-peer data transmission can enable one or more of the storage devices to support, for example, reliability, availability, and security (RAS) features, computational storage operations, etc., while reducing or eliminating the consumption of host interface bandwidth and / or interference with host services.

[0022] In some embodiments, one or more peer-to-peer channels may be implemented using wired, wireless, and / or other technologies. Any of these peer-to-peer channels may be arranged in a one-to-one (1:1), one-to-many (1:N), and / or many-to-many (N:N) configuration, or any combination thereof, and in a static and / or dynamic configuration.

[0023] According to the present disclosure, some embodiments may implement topology and / or characteristic discovery, connection establishment, message flow control, quality of service (QoS) configuration, and / or other features of storage devices that may exchange data via one or more peer channels.

[0024] Figure 1 An embodiment of a direct attached storage (DAS) system according to the present disclosure is shown. Figure 1 The storage system 100 shown may include one or more storage devices 102, which may be implemented as SSDs with non-volatile memory express (NVMe) interfaces. The storage devices 102 may be communicatively connected to a peripheral component interconnect express (PCIe) switch 104 via a PCIe link 105. The storage devices 102 may be connected to the PCIe switch 104 via a midplane 106, for example, using a storage connector such as a U.2 connector 108. The PCIe switch 104 may connect the storage devices 102 to a host 110 that may implement various storage control functions. The host 110 may include, for example, a redundant array of independent drives (RAID) controller 112 and one or more NVMe drives 114 for the storage devices 102.

[0025] The storage system 100 may also include a service processor such as a baseboard management controller (BMC) 116 to perform management and / or supervisory functions for the system 100. The BMC 116 may communicate with one or more of the storage devices 102 via a system management bus (SMBus) multiplexer 118, which may provide a management interface 120 to the storage devices 102 via the midplane 106 and storage connectors 108. The components of the storage system 100 may be housed in a chassis 122, a rack, a data room, or any other enclosure or space.

[0026] The storage system 100 may be accessed, for example, through a network interface 124 such as an Ethernet interface, which may provide access to one or more applications 126 and / or operating systems and / or file systems 128 through the Internet and / or any other cloud or network infrastructure 130. Although illustrated with PCIe switches and links, the interconnect fabric may be implemented using any storage interconnect such as Serial Advanced Technology Attachment (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), etc.

[0027] Figure 2 An embodiment of a remote direct attached storage (rDAS) system according to the present disclosure is shown. Figure 2 The rDAS system 132 shown in FIG. 1 may include one or more storage devices 134, which may be implemented as Ethernet SSDs (eSSDs) with non-volatile memory express over fabric (NVMe-oF) interfaces. The storage devices 134 may be communicatively connected to an Ethernet switch 136 via an Ethernet connection 137. The storage devices 134 may be connected to the Ethernet switch 136 via a midplane 138 using, for example, a storage connector such as a U.2 connector 141. The midplane 138 may be implemented as a single circuit board, a group of smaller distributed circuit boards, or any other configuration.

[0028] as Figure 1 The storage system 100 shown, Figure 2 The illustrated storage system 132 may also include a service processor such as a BMC 140 that may perform management and / or supervisory functions for the storage system 132. In some embodiments, the BMC 140 may communicate with the storage devices 134 via a PCIe switch 142 to provide a management interface 144 to the storage devices 134 via a midplane 138 and storage connectors 141. The components of the storage system 132 may be housed in a chassis 146, a rack, a data room, or any other enclosure or space.

[0029] The Ethernet switch 136 can enable a remote host 148 to access the storage system 132 through, for example, the Internet and / or any other cloud or network infrastructure 150. The remote host 148 may include one or more NVMe-oF drivers 152 for the storage device 134. The host may also include one or more applications 154 and an operating system and / or file system 156. Although shown with an Ethernet switch 136 and Ethernet connections, other network structures may be used in the storage system 132, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol / Internet Protocol (UDP / IP), Remote Direct Memory Access (RDMA), Fibre Channel, InfiniBand, etc.

[0030] In some embodiments, due to Figure 1 The storage device 102 shown in FIG. 1 may be connected to a local host 110, which may include a device driver 114 and one or more storage management features (such as a RAID controller 112), which may appear to function as a relatively cohesive and / or self-contained unit. In contrast, in some embodiments, Figure 2 The storage devices 134 shown in FIG. 1 may operate relatively independently of one another and may therefore appear to be a pool of SSDs to a local or remote host.

[0031] In some embodiments of the storage system, the implementation of one or more RAS features may involve one or more trade-offs between reliability, cost, performance, etc. For example, data redundancy-based data reliability techniques for recovering lost data from redundant data may involve one or more trade-offs between reliability level, redundancy, performance level, etc. In some embodiments, data reliability may be achieved through replication and / or backup copies of data, for example, using a RAID controller or storage system software stack in a centralized manner. Offloading one or more RAS features or a subset thereof to a storage device for distributed implementation may help reduce cost, complexity, performance bottlenecks, etc. In some embodiments, one or more peer-to-peer data transmission techniques according to the present disclosure may enable one or more storage devices to implement one or more RAS features without consuming host interface bandwidth, without involving local and / or remote hosts, etc.

[0032] In some embodiments of the storage system, the implementation of computational storage may also benefit from the use of peer-to-peer data transfers between storage devices according to the present disclosure. Computational storage may involve offloading computations from a central processing unit (CPU) to a storage device. In some computational storage use cases, one or more cost optimizations and / or performance benefits may be obtained by processing raw data at a storage device and sending the results or reduced data sets to a host. However, in some computational storage use cases, data may be striped or distributed across multiple storage devices for RAS purposes. Therefore, in some embodiments, computational storage may involve accessing data on more than one storage device to which computations have been offloaded. Therefore, computational storage may involve data transfers between storage devices.

[0033] Although data may be transferred between storage devices via host memory and / or switches in the host interface path, such transfers may consume host interface bandwidth, CPU attention, resources, etc. Moreover, transferring data between storage devices via host memory and / or switches in the host interface path may result in increased network infrastructure costs, energy consumption for data movement, performance bottlenecks, increased latency, suboptimal performance, etc. In some embodiments, and depending on implementation details, any number of these issues may be addressed by one or more sideband techniques for peer-to-peer data transfer between storage devices in accordance with the present disclosure.

[0034] In some embodiments according to the present disclosure, storage devices with peer data transfer capabilities can be discovered, a peer-to-peer network topology can be established, and the storage devices can exchange data using messages using, for example, a message transmission connection through one or more peer channels. In some embodiments, one or more peer data channel capabilities and / or data exchange technologies can be independent of one or more underlying storage architectures and / or protocols used between the storage device and one or more local and / or remote hosts (such as NVMe, NVMe-oF, DAS, RDA, etc.).

[0035] Figure 3 An embodiment of a storage system according to the present disclosure is shown, and the storage system may include two or more storage devices, and the storage devices may have peer-to-peer data transmission capabilities. Figure 3 The illustrated storage system 160 may include two or more storage devices 162 having storage interfaces 164 connected to a host interface 166 via one or more data connections 168. The storage devices 162 may also include a peer interface 170 connected to a peer bus 172.

[0036] The peer bus 172 may be used to implement one or more peer channels between the storage devices 162. In some embodiments, the peer bus 172 may refer to any or all of hardware components, software components, communication protocols, etc. that may be used to implement one or more peer channels. In some embodiments, any of the peer channels may be implemented using wired, wireless, and / or other technologies and / or combinations thereof.

[0037] Examples of wired technology according to the present disclosure may include one or more passive and / or active buses that may be coupled to storage devices via existing storage connectors and / or additional connectors. The bus conductors may be implemented on circuit boards, boards, strips, cables, harnesses, etc., and in some embodiments may be arranged for heat exchange, such as heat exchange with one or more connectors and / or sliding contacts.

[0038] Examples of wireless technologies according to the present disclosure may include radio frequency (RF) cellular interfaces such as 4G, 5G, etc., and any version of Wi-Fi, Bluetooth, etc. Examples of other technologies for implementing peer-to-peer channels according to the present disclosure may include optical communications such as laser communications (with or without waveguides such as optical fibers).

[0039] The storage interfaces 164, host interfaces 166, and data connections 168 may be implemented in any configuration, topology, etc. using any storage interconnect, network, protocol, connector, etc. (including SATA, SAS, NVMe, NVMe-oF, Peripheral Component Interconnect (PCI), PCIe, Fibre Channel, InfiniBand, TCP / IP, UDP / IP, RDMA, etc.). The components of the storage system 160 may be arranged in any physical configuration using, for example, one or more midplanes, backplanes, switchboards, motherboards, connectors, etc. In some embodiments, the components may be contained in one or more open or closed enclosures such as chassis, racks, rack groups, data rooms, data center buildings, etc. In some embodiments, the storage system 160 may be packaged in a storage chassis having any number of storage devices 162 (e.g., 24, 48, etc.).

[0040] In some embodiments, the storage system 160 may further include a management controller 174, which may include, for example, a service processor such as a BMC, which is configured to provide management and / or supervisory functions for the storage system 160. The management controller 174 may communicate with the storage device through any communication interface 176 including an SM bus (SMBus), a PCIe fabric, Ethernet, etc. The management controller 174 may communicate with a user, a host, a data center manager, etc. through any communication path including, for example, a dedicated network connection, a link to a host interface 166, etc.

[0041] In some embodiments, host interface 166 can be connected to one or more local and / or remote hosts. For example, in some embodiments, host interface 166 can be connected to a local host that can implement one or more storage control functions, such as RAID, erasure coding (EC), etc. In some other embodiments, host interface 166 can be implemented as a network switch or gateway, for example, to a remote host.

[0042] In some embodiments, one or more remote hosts, one or more storage devices 162 in the storage device 162 and / or the management controller 174 may include logic for implementing topology and / or feature discovery, connection establishment, message flow control, QoS configuration and / or other features to enable the storage device 162 to perform various functions using the peer channel, such as RAS functions, computational storage operations, etc.

[0043] For example, in some embodiments, topology discovery can be initiated and / or performed in a centralized manner by a host, a service processor such as a BMC, or the like. Alternatively, or additionally, or one or more storage devices can self-discover and / or establish connections with one or more other peer-capable devices in a distributed manner. As another example, storage devices can be arranged into peer groups and / or subgroups having various topologies and having various resources such as bandwidth, priority, etc. distributed between them. This configuration can be statically and / or dynamically performed by a host, a service processor such as a BMC, a service processor, or the like based on any number of policy parameters.

[0044] In some embodiments according to the present disclosure, peer-to-peer data transfer between storage devices can be used to implement data reliability techniques that use data redundancy to recover lost data using redundant data. For example, before data loss, two or more storage devices can exchange duplicate copies of data for backup purposes, and after data loss, data can be exchanged for recovery purposes with little or no intervention by the host. As another example, one or more storage devices can implement data striping across multiple devices.

[0045] In some embodiments, one or more storage devices may use peer-to-peer data transfers to implement computational storage and / or otherwise offload computational and / or bandwidth loads from one or more hosts and / or host interfaces. For example, in some embodiments, one or more storage devices in a group or subgroup of devices having a peer-to-peer channel configured between them may be designated to perform data compression, encryption, etc. on other storage devices in the group, subgroup, storage system, etc. As another example, in some embodiments, one or more storage devices may use a peer-to-peer channel to transfer intermediate computational results between devices for further computation without having to send and / or receive the intermediate results to and from a host.

[0046] Figure 4 An embodiment of a storage device according to the present disclosure that may have peer-to-peer data transfer capabilities is shown. Figure 4 The storage device 180 shown may be used, for example, Figure 3 One of the storage devices 162 shown. The storage device 180 may include a storage medium 182, a storage device controller 184, a storage interface 186, a peer interface 188, and a storage connector 190. The storage device controller 184 may be configured to control data transfer to the storage medium 182 and the interfaces 186 and 188. The storage device controller 184 may include peer logic 193, for example, for implementing any of the peer features disclosed herein, including topology and / or feature discovery, connection establishment, message flow control, QoS configuration, RAS functions, computational storage operations, etc. In an embodiment where the storage device 180 is implemented as an SSD, the storage device controller 184 may also include a flash translation layer (FTL) 194 and / or an NVMe manager 196.

[0047] In some embodiments, storage interface 186 may be implemented with any interface, protocol, etc. including SATA, SAS, NVMe, NVMe-oF, PCI, PCIe, Fibre Channel, InfiniBand, TCP / IP, UDP / IP, RDMA, etc. Storage connector 190 may be implemented with any suitable connector, for example, a U.2 connector that may be capable of accommodating different families and / or variations and / or combinations of storage protocols such as SATA, SAS, and NVMe.

[0048] In some embodiments, the peer interface 188 may be implemented with any communication interface for establishing one or more peer channels (including wired, wireless and / or other technologies and / or combinations thereof). In some embodiments, the peer interface 188 may be connected via a storage connector 190, for example, using one or more reserved, reallocated and / or new terminals (pins) on an existing connector design. In some embodiments, a new storage connector 190 may be designed to accommodate terminals for the storage interface 186 and the peer interface 188. In some embodiments, the peer interface 188 may be accessed via an additional connector 192 instead of the storage connector 190, or in addition to the storage connector 190. In some embodiments, for example, where the peer interface 188 may be implemented with wireless technology, the peer interface 188 may not use the attachment connector 192 or a connection via the storage connector 190.

[0049] In some embodiments, storage interface 186 and peer interface 188 can be integrated with each other. For example, in some embodiments, storage device 180 may include a dual-port Ethernet interface, wherein storage interface 186 can use one of the ports, and peer interface 188 can use another port. In some embodiments, peer interface 188 can be implemented at least in part as a virtual interface. In some embodiments, storage interface 186 and peer interface 188 can be implemented with substantially the same communication interface, wherein a peer channel can be implemented as a virtual channel by a switching structure for connecting storage interface 186 to one or more other storage devices and / or one or more local and / or remote hosts. Therefore, in some embodiments, a peer bus can be integrated with a switching structure (such as a network or interconnect structure) for a primary data path between one or more storage devices and one or more hosts.

[0050] Any or all of the storage device controller 184, storage interface 186, and peer interface 188 may be implemented in hardware, software, or any combination thereof.

[0051] Some example embodiments of systems, processes, methods, etc., which illustrate some possible implementation details according to the present disclosure are described below. These examples are provided for the purpose of illustrating the principles of the present disclosure, but these principles are not limited to these embodiments, implementation details, etc.

[0052] Figure 5 An example embodiment of a storage system having a storage device with peer-to-peer data transfer capabilities implemented using wireless technology according to the present disclosure is shown. Figure 5 The storage system 200 shown may include a Figure 1The DAS architecture of the embodiment shown in the figure may include some components of the DAS architecture, but the storage device 202 may include a wireless module (such as Figure 5 The system 200 may implement a wireless peer bus 206 that may have a 5G cellular transmit and receive (Tx / Rx) module as shown. Figure 3 Any number of features of the peer-to-peer bus 172 are shown.

[0053] In some embodiments, one or more of the 5G wireless modules 204 may be tuned and / or optimized based on the implementation details of the storage system 200. For example, if the system is packaged in a server chassis, the wireless modules 204 may be tuned for high bandwidth with low power and / or low latency due to the proximity of the modules in the chassis. As another example, if the storage drives 202 are more widely distributed in space, such as across an entire server rack, data room, data center, etc., the wireless modules may be tuned to make tradeoffs, such as trading off bandwidth to operate at the same relatively low power, or operating at increased power to maintain bandwidth at greater distances.

[0054] Although the storage system 200 is shown in the context of some specific implementation details (such as a DAS architecture with NVMe drives, 5G wireless technology, mid-plane, SMBus management interface, etc.), other embodiments using any cellular wireless module may be implemented with any other implementation details according to the present disclosure.

[0055] Figure 6 Another example embodiment of a storage system having a storage device with peer-to-peer data transfer capabilities implemented using wireless technology according to the present disclosure is shown. Figure 6 The storage system 208 shown may include a Figure 2 The rDAS architecture of the embodiment shown in the figure may include some components of the rDAS architecture, but the storage device 210 may include a Wi-Fi module (such as Figure 6 The storage system 208 may implement a wireless peer-to-peer bus 214 that may have a configuration similar to that of the Institute of Electrical and Electronics Engineers (IEEE) standard IEEE 802.3.11g module shown in FIG. Figure 3 Any number of features of the peer-to-peer bus 172 are shown.

[0056] In some embodiments, one or more of the Wi-Fi modules 212 may be configured similarly to the above with respect to Figure 5The storage system 208 is tuned and / or optimized in the manner described for the illustrated storage system 200. Although the storage system 208 is illustrated in the context of some example implementation details (such as an rDAS architecture with an eSSD drive, Wi-Fi wireless technology, a midplane, a PCIe-based management interface, etc.), other embodiments using any Wi-Fi wireless module may be implemented with any other implementation details according to the present disclosure.

[0057] Figure 7 An example embodiment of a storage system having a storage device with peer-to-peer data transmission capabilities implemented using optical communication technology according to the present disclosure is shown. Figure 7 The illustrated storage system 216 may include storage devices 218 having a peer interface implemented with a laser communication module 220. In some embodiments, one or more of the modules may have two ports 224A and 224B. The storage devices 218 and / or modules 220 may be arranged so that a port 224A on one module 220 may align with a corresponding module 224B on an adjacent module 220 (e.g., on a device in an adjacent chassis slot) and transmit data to and from a corresponding module 224B on an adjacent module 220, thereby forming a link 226 that uses laser beam encoding to exchange messages. One or more ports 224A and / or 224B may have a laser transmitter (Tx) and a receiver (Rx). Thus, the modules 220 may form a peer bus in a ring or chain topology that may have a communication interface similar to that of a wireless communication bus. Figure 3 Any number of features of the illustrated peer-to-peer bus 172. In some embodiments, one or more of ports 224A and / or 224B are capable of loopback operations, thereby enabling modules at either end of the chain to loop back messages not addressed to them, thereby forming a ring topology.

[0058] In some embodiments, one or more of the storage devices 218 may include a storage connector 228, such as a U.2 connector for connecting a storage interface in the storage device to a host interface for host traffic. Although the storage system 216 may be described in the context of some example implementation details such as a dual-port laser module 220 and a ring or chain topology, other embodiments using laser or other optical communications may be implemented with any other implementation details according to the present disclosure.

[0059] Figure 8 An example embodiment of a mechanical configuration of a storage system having a storage device with peer-to-peer data transfer capability implemented using wired technology according to the present disclosure is shown. Figure 8The system shown can include a storage device 232 having a storage connector such as a U.2 connector 234 (shown in phantom outline on the back of the storage device 232) and an additional connector 236. When the storage device 232 is inserted into a slot as shown by arrow 242 to establish a connection between a storage interface in the storage device 232 and a host, the storage connector 234 can be inserted into a corresponding socket on a midplane or other structure. A wired bus device 238 such as a top plate or ribbon cable can have a connector 240 that can be aligned with an additional connector 236 on a corresponding storage device 232. The additional connector 236 on the storage device 232 can connect a peer interface in the storage device 232 to an electrical trace on the bus device 238, thereby forming a peer-to-peer wired bus between the storage devices 232.

[0060] In some embodiments, one or more sliding contacts and / or spring-loaded pins 237 on the additional connector 236 can engage with one or more corresponding contact pads 239 on a corresponding connector 240 on the bus device 238, thereby establishing one or more electrical connections between the additional connector 236 and the bus device 238. In some embodiments, the position of one or more of the sliding and / or spring-loaded contacts and pads can be reversed. In some embodiments, the wired bus device 238 can be arranged, for example, across any or all slots in a chassis or other storage cabinet on the top side of one or more bays of a storage device 232, and the storage device 232 can have an additional connector 236 arranged along the top edge of the device.

[0061] Fig. 9 An example embodiment of an electrical configuration for a wired peer-to-peer bus for a storage system according to the present disclosure is shown. Fig. 9 The electrical configuration shown can be used, for example, with Figure 8 The mechanical configuration and / or bus arrangement 238 shown are used together. Fig. 9 In the illustrated storage system 230, the storage device 232 may include a peer interface 244 having dual ports 246A and 246B. The arrows may illustrate the flow of signals through electrical traces on the wired bus device 238 when the storage device 232 is inserted into a slot in the storage system. In some embodiments, one or more of the ports 246A and / or 246B may be capable of loopback operations, thereby enabling the interfaces at either end of the chain to loop back messages not addressed to them, thereby forming a ring topology.

[0062] In some embodiments, Figure 8 and Fig. 9 The electrical and / or mechanical arrangement shown may enable hot swapping of storage devices 232 in storage system 230 .

[0063] Fig.10 An example embodiment of a storage system having storage devices with peer-to-peer data transfer capabilities implemented using a passive wired bus connected through existing connectors according to the present disclosure is shown. Fig.10 The storage system 248 shown may include a Figure 1 Some components of the DAS architecture arrangement of the illustrated embodiment, but the storage device 250 may include a peer interface that may use existing connectors (such as Fig.10 One or more unused, undefined and / or redefined terminals or pins in the U.2 storage connector 256 shown in the figure. To avoid obscuring the drawings, the routing of the PCIe connection 105 and the management bus connection 120 is not shown on the midplane 252. In some embodiments, the electrical traces on the midplane 252 can be routed to form a passive wired peer bus 254 on the midplane. The passive wired peer bus 254 can be configured with any number and / or configuration of peer channels between storage devices 250 and / or groups and / or subgroups of storage devices. For example, in some embodiments, channels can be routed between adjacent storage devices 250 to form a chain and / or ring topology. In some embodiments, any other channel topology can be used, including a star topology, a multi-drop bus, and / or any combination thereof.

[0064] Although the system 248 is shown in the context of some specific implementation details (such as a DAS architecture with NVMe drives, U.2 connectors, and a peer bus on a midplane), in other embodiments, any other architecture, storage device type, connector, etc. may be used. For example, in some embodiments, the connection between the peer bus and the storage device may be made through a connector other than a U.2 connector, such as a connector for an SMBus interface or a new connector design that may be used to create new terminals (pins) for a peer data channel. As another example, in some embodiments, a passive peer bus may be implemented on a backplane, a switch plane, a motherboard, and / or any other device in a storage system.

[0065] Fig.11 An example embodiment of a storage system having storage devices with peer-to-peer data transfer capabilities implemented using an active wired bus connected through existing connectors according to the present disclosure is shown. Fig.11 The storage system 258 shown in FIG. 2 may include a storage system similar to Figure 2 2. Some components are arranged in the rDAS architecture of the embodiment shown in FIG. 2, but the storage device 260 may include a peer interface that can use existing connectors (such as Fig.11One or more unused, undefined and / or redefined terminals or pins in the U.2 storage connector 262 shown. To avoid obscuring the drawing, the routing of the Ethernet connection 137 and the management bus connection 144 are not shown on the midplane 264. In some embodiments, the peer interface connections at the connector 262 can be connected to a switch fabric to form an active peer bus between the storage devices 260. Fig.11 In the example shown, the switch fabric may be implemented as a PCIe switch 266, which may be located, for example, on a switch board in the chassis 146. PCIe links 268 between the PCIe switch 266 and the storage devices 260 may be routed through the midplane 264. An active peer-to-peer bus implemented with the PCIe switch 266 may be configured with any number and / or configuration of peer-to-peer channels between the storage devices 260 and / or groups and / or subgroups of storage devices.

[0066] Although the storage system 258 is shown in the context of some specific implementation details (such as an rDAS architecture with eSSD drives, U.2 connectors, and a switch fabric implemented with a PCIe switch), in other embodiments, any other architecture, storage device type, connector, etc. may be used. For example, in some embodiments, other types of switch fabrics may be used to implement peer-to-peer buses, including SATA, SAS, NVMe, NVMe-oF, PCI Fibre Channel, InfiniBand, TCP / IP, UDP / IP, RDMA, etc. As another example, in some embodiments, the switch may be located on the midplane, backplane, motherboard, and / or any other device in the storage system, and the connections from the storage device 260 and the switch fabric may be routed through any suitable means.

[0067] In some embodiments, a management controller, and / or one or more storage devices with peer data transmission capabilities may participate in a topology discovery process, which may involve discovering the presence of one or more storage devices with peer capabilities and / or configuring one or more peer channels between storage devices. In some embodiments, a host and / or a management controller that may include a service processor such as a BMC may initiate and / or manage a topology discovery process. In some embodiments, the slot or connector number where the storage device is located may be used as an identifier for the storage device. The discovery process may rely on one or more technologies for implementing a peer bus according to the present disclosure. For example, in an embodiment with a wireless peer channel, various topologies such as 1:1, 1:N, N:N and / or combinations thereof may be formed under the control of a host, BMC and / or storage and / or data center administrator. In some embodiments, such as using optical (e.g., laser) and / or chain hardwire topologies, a peer channel may have an inherent 1:1 ring or chain topology, although such topologies may still be changed and / or combined to create more complex topologies.

[0068] In some embodiments, in the absence of a topology determined by a host or a service processor such as a BMC, one or more storage devices can self-discover other storage devices with peer capabilities and establish connections. In some embodiments, the storage device may include one or more default parameters for the peer connection establishment process. In some embodiments, after the storage system is powered on, one or more storage devices in the system may broadcast their identifiers via a peer bus. One or more other peer storage devices may record these broadcast identifiers and / or use the broadcast identifiers to form various topologies, for example, 1:1, 1:N, N:N, and / or combinations thereof in ascending or descending order of slots. In addition, topologies and / or one or more groups and / or subgroups may be dynamically created by a host, a management controller, and / or one or more storage devices.

[0069] In some embodiments, one or more QoS parameters may be established for peer-to-peer data transfers between one or more storage devices, groups and / or subgroups of storage devices, etc. The QoS parameters may be established by one or more hosts, management controllers, storage devices, etc. For example, in a storage system having a designated storage device that may perform services such as compression, encryption, etc. for other devices, the designated device may have a default peer-to-peer bandwidth configuration for the service (i.e., a device-based QoS configuration), which may be overridden by a host, which may set a different bandwidth for the purpose of the service (i.e., a host-based QoS configuration).

[0070] QoS parameters may include priority, resources, bandwidth, latency, and / or other performance settings. In some embodiments, different QoS parameters may be applied to different groups or subgroups of storage devices. For example, within a chassis or rack of storage devices, a first group of storage devices may belong to an erasure coding (EC) domain involving more certain communications, and therefore, the first group may have higher and / or guaranteed QoS parameters, such as bandwidth, latency, etc. In contrast, a second group of storage devices may calculate checksums, and / or perform other background data verification tasks that may not involve real-time requirements, and therefore the second group may have lower QoS parameters. In some embodiments, one or more groups and / or subgroups may have one or more QoS parameters that may be tuned or optimized for peer communications compared to host services.

[0071] In some embodiments, one or more QoS parameters for one or more storage devices, groups and / or subgroups of storage devices, etc. may be based on one or more policy parameters. Examples of policy parameters may include application type, application identifier, namespace identifier, host identifier, logical block address (LBA) address range, non-volatile memory (NVM) set identifier, NVMe submission queue identifier, completion queue identifier, flow identifier, Ethernet MAC identifier, TCP / IP address and other transport / network parameters, as well as general parameters (such as the number of storage devices in a chassis, rack, etc., date, time, etc.).

[0072] In some embodiments, flow control and / or other reliability techniques may be implemented in one or more peer channels between storage devices according to the present disclosure. For example, confirmation / non-confirmation (Ack / Nack) opcodes may be used in messages between devices. For example, after a peer channel is established between a group of storage devices, the Ack / Nack opcode may be useful. If multiple devices in a group attempt to send data to a single destination device, the destination device may only accept a portion of the data sent, depending on resources such as buffer memory that may be available to the destination device. The use of the Ack / Nack opcode may enable the device to determine which parts of the data sent are accepted, and which (if any) devices may need to resend the data. As another example, the message length and / or message sequence number may be used in a completion entry. For example, in some embodiments, when a source device sends data to a destination device, the destination device may confirm the receipt of the data by placing the message length in the completion entry. As another example, in some embodiments, the message may include a sequence number so that when the message arrives at the destination device, the destination device may determine whether there are any lost messages that can be resent.

[0073] In some embodiments, data messages may be used to transfer peer-to-peer data between storage devices. Fig.12 An example embodiment of a message format according to the present disclosure is shown. Fig.12 The illustrated embodiment may include a header portion and a data portion. The header may include identifiers for the transmitting device and the receiving device. In some embodiments, the transmitting and / or receiving device may be identified by a slot ID in a rack or other housing. In some embodiments, the header may also include additional fields such as message length, message sequence number, Ack / Nack opcode, data protection field (such as a cyclic redundancy check (CRC) code of the header) and / or data payload, etc.

[0074] In some embodiments, for one or more storage devices, the peer-to-peer data transfer capability according to the present disclosure can be implemented as an optional feature so that if the device is installed in a system without peer-to-peer data transfer capability or installed with other storage devices without peer-to-peer data transfer capability, the device can be configured as a standard storage device for default behavior. For example, the peer-to-peer interface may not affect normal data traffic between the storage device and the host, as well as normal host command arbitration, execution, completion, etc. Therefore, a device with peer-to-peer data transfer capability according to the present disclosure can coexist with other storage devices. For example, in some embodiments, the peer-to-peer data transfer capability can be enabled and / or disabled under the control of the host and / or management controller.

[0075] Fig.13 A data flow diagram illustrating an embodiment of a method for operating a storage system in accordance with the present disclosure in which storage devices may exchange data through one or more peer-to-peer channels. Fig.13 The illustrated method 270 may be implemented, for example, using any of the storage systems described herein. Fig.13 In the method 270 shown, the storage device 272 has a storage interface 274 and a peer interface 276. The storage device 272 can be divided into a first group of storage devices 278 and a second group of storage devices 280. The first group 278 can also be divided into a first subgroup 282 and a second subgroup 284. Fig.13 As shown by the arrows in , peer-to-peer traffic can flow between storage devices within each group and / or sub-group. Host 286 can have a host interface 288, which can be connected to storage interface 274 on storage device 272 to enable host traffic to flow between the host and the storage device.

[0076] Fig.14 An embodiment of a computing system according to the present disclosure is shown. For example, Fig.14The system 300 may be used, for example, to implement any or all of the methods and / or apparatus described in the present disclosure. For example, the system 300 may be used to implement a host, a service processor such as a BMC, a storage device controller in a storage device, etc. The system 300 may include a central processing unit (CPU) 302, a memory 304, a storage device 306, an interconnect interface 307, a user interface 308, and a network interface 310. In different embodiments, the system may omit any of these components, or may include a copy or any additional number of any of the components, as well as any other type of component, to implement any of the methods and / or apparatus described in the present disclosure.

[0077] In some embodiments, the interconnect interface 307 may be implemented with any type of general purpose or storage interconnect such as SATA, SAS, NVMe, PCI, PCIe, etc.

[0078] CPU 302 may include any number of cores, caches, buses and / or interconnect interfaces and / or controllers. Memory 304 may include any arrangement of dynamic and / or static RAM, non-volatile memory (e.g., flash memory), etc. Storage device 306 may include a hard disk drive (HDD), a solid state drive (SSD), and / or any other type of data storage device or any combination thereof. User interface 308 may include any type of human-machine interface device, such as a keyboard, mouse, monitor, video capture or transmission device, microphone, speaker, touch screen, etc., and any virtualized or remote version of such a device. Network interface 310 may include one or more adapters or other devices to communicate via Ethernet, fiber channel, InfiniBand, TCP / IP, UDP / IP, RDMA, Wi-Fi, Bluetooth, or any other computer networking arrangement to enable components to communicate via physical and / or logical networks (such as intranets, the Internet, local area networks, wide area networks, etc.).

[0079] Any or all components of the system 300 may be interconnected via a system bus 301, which may collectively refer to a variety of interfaces including a power bus, an address and data bus, a high-speed interconnect such as Serial AT Attachment (SATA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (PCI-e), a System Management Bus (SMBus), and any other type of interface that may enable components to work together locally at one location and / or distributed between different locations.

[0080] The system 300 may also include various chipsets, interfaces, adapters, glue logic, embedded controllers (e.g., programmable or non-programmable logic devices or arrays), application specific integrated circuits (ASICs), embedded computers, smart cards, etc., which are arranged to enable the various components of the system 300 to work together to implement any or all of the methods and / or devices described in the present disclosure. Any component of the system 300 may be implemented in hardware, software, firmware, or any combination thereof. In some embodiments, any or all components may be implemented in a virtualized form and / or in a cloud-based implementation with flexible resource provisioning, such as within a data center, or across multiple data centers.

[0081] The operations, methods, algorithms and / or functions described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in one or more software modules executed by a processor, or in a combination of the two, which are included in the system 300. If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a tangible, non-transient computer-readable medium. The software module may reside in a random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium. Any system disclosed herein or its components or portions thereof may be implemented as part of a software stack of a larger system (e.g., a storage system or other larger system). Any system disclosed herein or its components or portions thereof may be implemented as its own software stack.

[0082] The embodiments disclosed herein may be described in the context of various implementation details, but the principles of the present disclosure are not limited to these or any other specific details. Some functions have been described as being implemented by certain components, but in other embodiments, the functions may be distributed between different systems and components in different locations and have various user interfaces. Certain embodiments have been described as having specific processes, steps, combinations thereof, etc., but these terms may also include embodiments in which specific processes, steps, combinations thereof, etc. may be implemented with multiple processes, steps, combinations thereof, etc., or embodiments in which multiple processes, steps, combinations thereof, etc. may be integrated into a single process, step, combination thereof, etc. References to components or elements may refer only to a portion of a component or element. The use of terms such as "first" and "second" in the present disclosure and claims may be merely to distinguish the things they modify, and may not indicate any spatial or temporal order unless otherwise clear from the context. Mentioning the first thing does not imply the existence of the second thing. In addition, according to the inventive principles disclosed in this patent, the various details and embodiments described above may be combined to produce additional embodiments.

[0083] Since the inventive concepts of this patent disclosure may be modified in arrangement and detail without departing from the inventive concept, such changes and modifications are intended to fall within the scope of the appended claims.

Claims

1. A method for storing data, comprising: transmitting data between the host and the first storage device through the first storage interface; transmitting data between the host and the second storage device via a second storage interface; transmitting data between the first storage device and the second storage device via a peer-to-peer bus; wherein data is transmitted between the first storage device and the second storage device using the peer-to-peer bus based on a quality of service QoS parameter; wherein the QoS parameters are based on one or more policy parameters; and The policy parameters include at least one of the following: Application type, application identifier, namespace identifier, host identifier, logical block address LBA address range, non-volatile memory NVM set identifier, non-volatile fast memory NVMe submission queue identifier, completion queue identifier, stream identifier, Ethernet MAC identifier, and transmission control protocol / Internet protocol TCP / IP address and other transmission / network parameters.

2. The method of claim 1, further comprising discovering a topology of a peer-to-peer channel. 3 . The method of claim 2 , wherein the discovery of the topology is performed at least in part by one or more of: the host, a management controller, or at least one of the storage devices.

4. The method of claim 1, further comprising configuring the QoS parameters for the first and second storage devices.

5. The method according to claim 4, wherein: The QoS parameters are configured for the storage device by one or more of: the host, a management controller, or at least one of the storage devices.

6. The method of claim 4, wherein the QoS parameters include one or more of priority, resource, or performance level.

7. The method according to claim 1, further comprising: transmitting data between the host and a third storage device via a third storage interface; as well as The first, second, and third storage devices are configured into one or more peer groups. The method of claim 7 , wherein the peer group is created dynamically.

9. The method of claim 7, wherein the peer group can be configured for one or more of one-to-one, one-to-many, or many-to-many communications.

10. The method of claim 1, wherein at least one of the storage devices performs at least one of reliability, availability, or security functions autonomously from the host.

11. A storage system comprising: Host interface; a first storage device having a first storage interface coupled to the host interface; a second storage device having a second storage interface coupled to the host interface; a peer-to-peer bus coupled between the first and second storage devices; wherein the first storage device and the second storage device are configured to transmit data between the first storage device and the second storage device using the peer-to-peer bus based on flow control and quality of service (QoS) parameters; wherein the QoS parameters are based on one or more policy parameters; and The policy parameters include at least one of the following: Application type, application identifier, namespace identifier, host identifier, logical block address LBA address range, non-volatile memory NVM set identifier, non-volatile fast memory NVMe submission queue identifier, completion queue identifier, stream identifier, Ethernet MAC identifier, and transmission control protocol / Internet protocol TCP / IP address and other transmission / network parameters.

12. The storage system according to claim 11, wherein: The first storage interface is coupled to the host interface via a first storage connector; The second storage interface is coupled to the host interface via a second storage connector; and The peer-to-peer bus is coupled between the first storage connector and the second storage connector.

13. The storage system of claim 11, wherein the peer-to-peer bus comprises a switch fabric.

14. The storage system of claim 11, wherein the peer-to-peer bus is located at least partially on one of a midplane, a motherboard, or a switch plane.

15. The storage system according to claim 11, wherein: The first storage device is coupled to the peer bus via a first peer connector; as well as The second storage device is coupled to the peer-to-peer bus via a second peer-to-peer connector.

16. A storage device comprising: Storage media; a storage device controller coupled to the storage medium; a storage interface coupled to the storage device controller; a peer interface coupled to the storage device controller; wherein the storage device is configured to transmit data using the peer interface based on a quality of service QoS parameter; wherein the QoS parameters are based on one or more policy parameters; and The policy parameters include at least one of the following: Application type, application identifier, namespace identifier, host identifier, logical block address LBA address range, non-volatile memory NVM set identifier, non-volatile fast memory NVMe submission queue identifier, completion queue identifier, stream identifier, Ethernet MAC identifier, and transmission control protocol / Internet protocol TCP / IP address and other transmission / network parameters.

17. The storage device of claim 16, wherein the peer interface comprises one or more of a wired interface, a wireless interface, or an optical interface.

18. The storage device of claim 16, wherein the storage interface comprises a dual-port interface.

19. The storage device of claim 16, further comprising a storage connector coupled to the storage interface, wherein the storage connector is coupled to the peer interface.

20. The storage device of claim 19, further comprising an additional connector coupled to the peer interface.

Images